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I. OBJECTIVE 


GE Tuer en AE ne af E 


A computer system is a complex and dynamic mixture of 


hardware, software and human resources that interact 
together zc provide a service. Barly computers had slow, 
expensive and limited ressources. The need to increases the 


performance of the system to best take advantage of che 
these expensive limited resources pecame a challenge and i 


necessity. On a batch system performance evaluation was fa 


ri 


less complicated then on the multi-task operating envircn- 
ments of the third generation computers. Computer 
Performance Evaluation (CPE) cf a computer system can easily 
become very expensive in many areas: system resources, 
Manpower, and financial. Even after large expenditures in 
all of these areas, the results ars usually hard to inter- 
peret or understand and may lead to meaningless conclusions 
mire can actually be of little value in providing improved 
performance. 

In order to achieve successful and meaningful CPE it is 
necessary to use Macro Computer Performance Evaluation, 
which basically means to design, davelop and implement a 
plan or framework to guide and direct CPE efforts. This 
involves the coordination of both a Computer Performance 
Management (CPM) strategy and a technical evaluation (meas- 
urement effort) strategy. Obviously, a CPM strategy without 
atechnical evaluation strategy for gathering measurements 
will not produce any useful results. On the other hand, a 
technical evaluation OE CES Computer Performance 
Evaluation, strategy without a well thought out  CPM will 
most likely lead to the situation of questionable or useless 


results mentioned earlier. 
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Popa cve Of this thesis is to provide a guide fer 


buiiding the foundation for both the CPM component and th 


a 


CPE technical evaluation component for a computer perfora- 
ance program. The effective use of computer resources will 
te emphasized and some technical details and expertises 
required to carry out the measurement and analysis phase of 
CPE will be introduced. Emphasis will be placed cn che idea 
that it is not necessary to get involved in the more sophi- 
cated and costly CPE tools such as software monitors an4 
hardwar monitors in order to obtain noticeable performance 
enhancements. 

It is acknowledged that guides for CPE are no« new and 
much research has been completed producing abundéent inforna- 
ton. However, the general approach of past research tends 
to view performance issues from the internal component 
perspective cf the computer system and the emphasis is on 
the efficient use of limited, expansive hardware and soft- 
ware resources. This traditional view caused most CPE 
efforts to rely upon expensive and sophisticated hardware 
end eer ware Wonitoring tools. Today, hardware is getting 
less and less expensive and is therefore no longer consid- 
ered a critical limited resource. More and more computer 


power is given to the end user and the emphasis of CPE i 


+ 


۲ 


switching from the efficient use of the system to effectiv 


ID 


(U 


utilization of the system. This quide will place mor 
emphasis on the effective use of the system instead of the 
"raditionaly quantatively oriented, efficient use of the 
computer system. The hardware maybe working very effi- 
ciently at transferring data from secondary storage to 
primary storage, but the user may not have blocked the 
records. Much more effective use can be made of the 
computer resources by blocking the file, therefore 


increasing tke performance of the computer. 
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The traditional concept of computer performance =valua- 
e20n must broaden it's scope from a microscopic interna 
resource concentraticn to a morae macroscopic  ex-*szra 
resource, total system evaluation CPM/CPE effor+. 240 9 
and other indirectly coupled rescurces can have a signifi- 
cant impact on the performance of the system. Many computer 
facilities require tremendous amounts of an organizatio 

۰ 


resources to implement and operate. Performance or «hese 
Ja 


th 
O 


computer systems must be managed to avoid critical per m 
ance shortccmings that jeopardize the needs of the organiza- 
tion. Traditional CFE tools and methods as weil &s new ones 
wi support the CPM effort, 

The guide is intended fer both managers and technical 
personnel who are concerned about computer performance 
and/or are about to become involved in a CPE endeavor. 
Managers will be able to obtain a fael for the scope and 
depth of effort as well as the support required to perform 
E effectively. Inexperienced technical’ people will be 
able to find out what tools and methods are avaliable for 
pericrming measurements. 

A second objective of this thesis is to explore thea 
impact new trends and endeavors in the field will place on 
puescur-ent concept of CPE. It appears that current CPE 
capability will not be adequate for networks (individual 
systems connected by communication links), array processors 
(systems containing thousands of processors), or Data Base 
Management Systems (CBMS). Ideas in some of these areas are 
presented in Chapter VIII "Current and Future Technology and 
Performance". 

Appendix A, "Sources of Information", is intended to 
provide both managers and technical personnel with  addi- 
tional sources of information in their areas of interest. 
Appendix B, "Examples of Performance Measures" provides a 
listing of some of the more common performance measuremerts 
and arranges the measurements into five groups: 
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1. 
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4. 
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Quantity of work performed 
Quality of service 
Component utilization 
Underlying factors 
Workload characteristics 
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II. INTRODUCTION 


"The first computer had no more than two cperarional 
programs before the idea occured to insert check £ O 
the coding t5 obtain a measure of how far the  prcgram had 
gotten befor2 it, or the machine, ceased operation" 
[iRef. 1]. Surpriszngiy, most manufacturers of computer 
systems as well as managers and users of these systems have 
not accomodated this interest in performance to any appreci- 
able degree. ACCorEding to Olson [Ref. 2] "Data processin 
activities have been reluctant in the past to address firn 
performance requirements." Tots Peluctance on the part oí 
ell personnel associated with the computer may be attributed 
bare to the feeling that if no set standard exists, they 
can never be held accountable for not meeting it. The 
preliminary investment in time and resources required by 
most computer performance efforts can also be a major factor 
in postfoning a sericus interest in performance until a 
crisis erupts demanding immediate performance attention. 
Some simple architectural modifications could significantly 
aid the hardware monitoring effort; this is discussed 
further in the section on hardwars monitors. One of the 
oldest indicators of performance is the CPU busy (or CPU 
Wait) light which is usually located on the front console of 
the computer. However, the CPU busy light was not intended 
for performance observation, but rather for maintenance 
applications and it is hard to measure the number of micro- 
seconds of idleness by examing *he CPU busy light. 

Morris also points out that “the two most portant and 
effective tools for evaluation of computer performance are 
available free in nearly 2very computer installation. They 


are visual inspection and common sanse". For example, ٤ 
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the busy light is constantly on (meaning CPU is busy) while 


the program is 2xecuting, zhen the "program could be CPU 
Found. Perhaps program tuning could be used to improve the 
performance of the progran. is che COU ighesis constantly 


Off and the disk units are "dancing about" tne computer room 
floor, this indicates that the programs executing are I/O 
intensive, and perhaps the paging algorithm cculd be 
adjusted or the blccking buffer size should be changed. 
Blocking buffer size is an important performance issue and 


Memomceuesed in detail in a later section. This could also 
indicate that there may be very little "overlap" achieved 
between the CPU and I/O. Sone fallacies that can result 


from the use of these tools are diszussed in Chapter IV "CDE 
Tools and the System Life Cycle. 

Another example, presented at a computer performance 
conference, concerns the degraded performance of a system 
between the hours. of 11 A.M and 1 P.M. Snortly after the 
installation of the new version of the operating system, the 
System Administrator (SA) began receiving compaints about 
the increased turnarcund time for jəDs for the period ٥ -وع‎ 
sponding to the two hour period in which most people took a 
lunch break. The SA was very puzzləd since prior to the 
installation of the revised operating systen, turnaround 
time on the »ld operating system usually improved during 
this period. 

After several days of inquiring about new jobs run, and 
having uncovered no new results, the SA called the vendor's 
System analyst out of frustration. The SA mentioned the 
possible need for a software or hardware monitor to the 
analyst. The analyst suggested that they first look ata 
dump of the accounting data for a period shortly before and 
after the installation of the revised operation system. 
Several new "system programs" began to dominate the job 


queue shortly after the installation of the revised version 
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of the operating systsm and during the 2 hour 


" 
D 
rj 
i Jo 
O 
نام‎ 


neh 
a 


Uu 


Zune! 
Upon locking at ths source file of these progranm 


(t 
e 
D 


, 


O 


ENS cedrIScOVe-red that che majoric-y of zaen were f Lee 


(D 
Fs 
£3 
fu 


"package" included in the revised version of 


cr 
ng 

iD 

01 


he acing 
= 


system compliments of the vendor. This packag ned ouz 
C 


O 
a - 

to be a collection of games used by “he vendor for prono- 
tional purposes. The employees of the computer installarior 
had renamed the games using systen-like names, +o avoid 
being easily detected, and were playing the games durina 
Munch tie. 

This example demonstrates the vendor systems 

E CoNEcUG:ce of using che two most important CP 
visual inspection and common sensa. Common sens 
also prevent any further, more detailed investigatio 
Meese first casual “visual inspection" without takin 


tims to develop a sound computer performanca management 
en 


plan. To proceed without such a plan would be expensive, 
wasteful and most likely produce unsatisfactory, possibly 
misleading results. A common sense initial approach would 


use the 80-20 rule, i.e. 80% preparation, and 20% perspiza- 
ON. Regrettably, the most common approach is depicted in 
Ste 2.1 Obtained from Bell (Ref. 3]. 


I 
| 
A 


Choose Perform Wonder 
a Some What To Do 


Monitor Measurement with the Data 
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Figure 2.1 À Common Approach to Performance Efforts. 
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CPE is usually first considered after a system has beer 
used for a period of time and becomes unable to provide 
satisfactory performance to either a specific program cr t3 
the whole system in general. Aczually, CPE could 55 
Berzormed at all phases of the life cycle of a computer 
system, which typically includes the following phases: 


e Procurement Phase 
e Installation Phase 
e Operation Phase 


e Transition Phase (tc a new systen) 


Due to economic reasons, formal See ente Leo. ¡(OPENS 
usually restricted to large mainframes and super computers 
supported by large enough budgets to afford this effort. 
Minicomputer systems usually recaivs inexpensive, Very 
informal CPE in the first two phases where the predominate 
goal is to buy as much hardware (main memory and secondary 
storage-disk capacity) as their procurement budget will 
Support from the vendor who has the best reputation for 
quality system software and software development tcols. 
This approach is most likely motivated by the fact that che 
most Expensive item in a computer system's long term budget 
is software development costs, not hardware costs. However, 
the software is extremely dependent upon the hardware 
purchased, and relies upon a certain level of performance 
from this hardware. If the issue of hardware performance is 
not adequately addressed during th2 procurement phase, all 
the expensive software developed will be of little value on 
a systen 602۶ ۹۶ supply the required level of 
per formance. 

Once the decision has been made to attempt CPE on a 


system, a CPM program should be initiated, and it's first 
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task should be to establish a strategy or define a ss of 
CPE objectives and their scopes. This is covered in more 
detail in Chapter III "Computer Performance Managemen+". 

When attempting to locate potential areas and botzle- 
necks affecting system performance the two most inportant 
and effective tools for evaluation: visual inspection and 
common sense shculd be used. Most of today's systems have 
accounting packages already installed -and are collecting 
valuable CPE information, mainly about the workload. A 
visual inspection of the accounting data will give a clue to 
what kinds and types of jobs use the most resources on the 
systen. Use of software monitors and hardware monitors 
should be put off until they are absolutely needed and can 
be proved tc be cost effective. Once a hardware monitor is 
introduced into the CPE effort the expertise and technical 
resources required to support the hardware monitor in order 
to gain neaningful and beneficial  resul-s will most 
certainly become one of the most expensive items in the CPE 
budget. Chapter IV CPE Tools and the System Life Cycle" and 
Chapter V "Choosing a HWonsor-cHampdwes5elfor Softwars?" 
provides information in this area. 

Personal computers (PC's) and networks are evolving and 
gathering respect for their ability to solve a wide range of 
problems. This is an indication that decentralized tech- 
nology and end user computer power is on the rise. This 
means that people no longer have to go to the computer in 
the sense that they must leave their office to obtain the 
services of the computer. Advancss in computer communica- 
tion technology has made it possiblə to take the services of 
the computer to the users in their office. Decreasing costs 
as a result of advances in integrat]2d circuit technology has 
made it possible to provide local computer capability at 
reasonable costs. Performance of the "system" then, 


concerns global as well local considerations. 
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Our traditional views of CPE, as well as our methods, ars 
being challenged by the emergence of new and expanding 
computer technology which demands new methods and approachss 
to perform meaningful CPE. On the other hand, there are 
easy to use performance improvements, presented in Chapter 
VII "Improving Performance Without Monitors", that do not 
receive wide spread use, but could significantly improve ths 
performance of a system. Chapter VI "Some Common Seısa 
Performance Fallacies" zeinforces the idea that hypotheses 
about performance issues must always be validated by 


measurements. 
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III. COMPUTER PERFORMANCE MANAGEMENT 

Before any professional carpenter reaches for his meas- 
uring tape to cut a peice of lumber, he will always corsult 
۶۰۰٦٦6 6210-۰۴۰ or plan first. This is usually a nen-al 
exercise in most cases, for he has previously gone over the 
actual plan many times in great detail, but he also retur 
to it frequently tc review critical neasuremerts. The 
important issue is that he works from a plan. This sa 
time, coordinates his activities toward a predetermined g 
and avoids waste. The above approach should also be re 
vant fcr anyone iavolved in CPE. 

Ir FIPS PUB 49 LERef. 0], a CPM program is defined as 
"any structured effort, in hcuse or otherwise, to measure 
and evaluate the performance of a computer facilit in 
support of established management goals and objectives". In 
additicn, the CPM program should strive to maintain tha 
computer system at the highest performance-to-cost ratio and 
provide management with reports on the status of the 
computer system. These reports should be simple, short, and 
easily understandable by management. 

The immediate objective of a CPM strategy is not to 
trcubpleshoot the system or to do system tuning, but rather 
to maintain, or in some instances, regain control ofa 
complex, costly and critical resource through a quantitative 
and qualitative understanding of how that resource performs 
and of the alternatives that are avaliable to mak it 
perform more effectively and efficiently. TRUS, CPM is 
obligated to serve management, who initially approved 
funding for the computer facility resource, and to the users 
who must use the system to meet the organization's estab- 


lished goals and objectives. 
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A. GETTING STARTED--THE FIRST STEP 


Management must realize that computers directly affect 
an Organization's bottom line. This GCE is ezp2ct:4 to 
increase substantially in the future, due to the unusual 


nature or the computer products market where physical size 


of the systen is decreasing, Capacity and capability are 
increasing and prices are decreasing. Thus, the mainfrane 


computer of yesterday, usually found in a computer room away 
from people, is now showing up in small packages on the «ops 
of desks in the office enviornment. 

Spsnetrr [ Ref. 5] defines the roles and responsibilities 
of the information-systems manager who will most likely rely 
upon the CPM group to provide most of the information he 
neeäs. Therefore, the CPM manager must be aware of this 
role and be capable of supporting the Fol low2ng 


responsibilities: 


1. Enable the organization to understand, apply and 
Control internal forces, and properly utilize 
external forces, that shape a firm's computing envi- 
ronment. 

2. Apply proven management principles, particularly 
strategic planning, +o information-systems functions. 

3. Provide the organization with planning and develop- 
ment concepts and tools that effectively «enable the 
firm to develop and implement a corporate computing/ 
business strategy, and to provide a logical framework 
in which it can understand newest usage trends in 
information systems management and computer tech- 


nology. 


4. Provide the organization with a forum to exchange 
ideas and discuss opportunities with information 


systens specialists. 
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9. Comprehend the critical nature of ¿information systems 
2 ns In the context of overall و۶۹ و‎ 3 6222-71 


SUCCESS. 


A wide range of computer performance  measurments are the 
pzihary source of information for the above areas. 

In the aggregats, computer systems represen= the 
Borolery of the organization itself, iN thar, one Vital 
corporate resource--information--is provided to the other 
vital corporate rasource--people. Therefore, timely zale- 
pn p accurate information, effectively produced and disse- 
ninated, is a vital ccrporate resource. Ta ALSA L GOLE CBT 
must use CPE in providing this resource in the most əffi- 
cient and effective manner. 

Regrettably, one external reason for the existence of a 
CPM group in the Federal Government and the DoD is the 
extremely long procurement tize required to order new equip- 
ment. Typically, if the workload on a system becomes toc 
much and extensive enhancements or a new system is nesded, 
it wili take a minimum of one year to obtain the desired 
equipment using the extremely long and paperwork intensive 
ADP procurement process required by the Federal and DoD. 

On the positive side, steps must be taken to maintain the 
best system performance during this procurement time, which 
motivates the organization to implement CPM and CPE. 


1. Selecting CPM Team Members 


In order to develop a CPM plan it is necessary to 
select personnel to form a CPM team or group. cs pecus ol 
cally wise *o select at least one represenetive from each of 
the various departments that can directly influence the 
performance of the computer system in order to insute 
overall cooperation. It ais both politically wise and 


economically essential to include a representative from 
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upper management, since they traditionally are responsible 
tor allocating funds for such groups and can give the cpu 
Ei Ene needed Teccgnition and importance to sustain ar 
effective role in the organization. 

Each representative need not be a full tine active 
member of the group, but should remain aware of the main 
Seer Of the CPM effort. Pull “eze paresieips:ion Of all 
members is not very practical in a large organization with 
many departments, since it would tend to maka «he CPM group 


too large and possibly ineffective. Small organizations 


(0 


Esta have difficulty also, since it may not be possibles to 
dedicate many people to this effort on a full time basis due 
to lack of personnel. fe DOSS51513, ad ea ium 6 t coss": 
fulltime CPM personnel should be choosen, that are well 
Rude -euofrom "fire fighting" and the day to day disjoin= 
crisis situations which usually exist in a computer system 
work environment +o insure continuity in thse CPN effort. 

ime o ganization rorning a CPM group is part of 
the Federal Government or the Department of Defense "(DoD), 
two organizations can provide a great deal of help sinca 
both organizations are dedicated +2 helping in the area of 
gem and CPE. The Federal Government organization is known 
as FEDSIM (Fed2ral Automatic Data Precessing Simulation 
Genzer). See [ Ref. 6] for location and contact information. 
DuScIHg frem the organizatlon's brochure: "FEDSIM provides a 
central source for computer simulation services sc that 
individual agencies will not need to develop independent 
capabilities to use advance technigues of computer perforn- 
ance measurement and evaluation. This will allow ali agen- 
cies to have access to powerful techniques on a cost 
reinbursement basis without incur ing high individual 
start-up costs in tige, money and expertise". 

FEDSIM policies are established by a Joint Policy 


Committee made up of representatives of GSA, Air Force, the 
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2) Analytical Routines 

FEDSIM was established by tha GSA and is operated by 
the Air Force at the request of GSA. 

The second organization is the Navy Regional Data 
Automaticn Center (NARDAC), Pensacola. See [ Ref. 7] for zhe 
complete organization title and location. This organization 
has the responsibility to provide tools, methods, and proce- 


dures EOE Compu ter Performance Evaluation/Ca pacity 


and consultation services to Navy activities as required. 
eeu EO S. B. Olson (Raf. 2] "The first fact >c 
Gone to light was that research indicated that effective 
Performance Management functions ware simply not being 
eecempecshed within Navy data processing activitiss. In 
fact, evidence suggested that throughout industry as well, 
theze was relatively little well organized and systemat 
Performance Evaluaticn or Capacity Planning baing done." 

Cre of the primary reasons perceived by Olson as 
hampering good ¡performance and capacizy management was the 
"general lack of a stand-alone specialized group whose 

rimary responsibility was the performance management 

ESSO. Typically, it was found, when studies were made or 
data collec:ued, the effort was usually assigned, perhaps by 
default, "2ہ 560 ۶ہ مہ۰‎ group 3E day-to-day. production 
people at the data processing center. 

Obviously, as pointed out earlier, in a small grcup, 
this problem would be hard to eliminate without a noticable 
loss of Support in other areas. However, it does indicate 
that for larger groups, management was either not involved 
or involved but unaware of the potential benefizs of a wall 
organized CPM program and therefore treated the CPM efforts 
with low or no priority when "fires" occured. One possible 
solution to this problem is to educate management as to the 
potential benefits of a CPM program and assign a priority to 
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CAE Ys high encugh so that it's objectives car be 
accomplishec uninterrupted. Again, it is suggested chat az 
least one person, possibly rotating Vene ole amongst 
different members in the group, be zada the continuing full- 
time active member in the CPM group. This should ensure 
thaz the CPM effort does not stall, but continues tc meet 
its objectives. 


2. Training The CPM Tean 


Once selected, the CPM team must become well 
acquainted with many ill defined areas concerning CPM and 
CPE such as: present system workload, projected system work- 
Boa, current computer capacity, and most importantly the 
objectives and the goals of the organization and how the 
computer center will aid in meeting then. à first step for 
Management training for any CPM group which is a part of the 
Federal Government would be to contact either FEDSIM or 
NARDAC. Potenza l management training for non Federal or 
DoD CPM groups would have to rely on firms in rivete 
industry that provide consulting services in the CPM and CPE 
fields. One such firm, founded by M. F. Morris coauthcr of 
[Ref. 1]. is Management Advisory Service, located in 
Washington DC. Mr. Morris was with FEDSIM in the past. A 
government agency ‘that does GEE: Priced information 
relating to CPM and CPE is the National Bureau of Standards, 
see (Ref. 8]. 

One specific source that provides extensive informa- 
LuonMonecapacity planning is "Capacity Planning: A State of 
the Art Survey", [Ref. 9]. This reference presents a very 
đetailed discussion of capacity planning and provides many 
recent references. Other sources of information, including 
CPE/CPM users groups and conferences, which usually have 
published proceedings, can be found in the Appendix listing 


Sresoueeces Of information. 
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B. CPM RESPONSIBILITIES 


Mecomdjing to FIPS PUB 49 [Ref. 4] CPM responsibili-iss 


will deal with at least four major areas: 
e User Requirements 
e System Resource Managenent 
e Communicating With Upper Management 
e Vendez Relations 


Prior to beginning a discussion in the above areas it aust 
be pointed out that these areas can help an organiz 

meet both its short and long range goals and objective 
iieomerem Group should obtain a written description or state- 
Ben. Of Management's ideas and 2xpectations of how the 
computer system and its related resources are responsible 
for helping to meet the organizations long-term and short- 
term goals and objectives. This may require repeated 
discussions in order to fully understand exactly what 
Management means and expects. In many cases management is 
nost likely poorly informed about how well the computer 
facility is actually mesting the organizations goals and 
objectives or what the possible effect of increasing the 
current workload will have on the performance of the systen. 
It is important that management is informed about it's 
computer facility. It is also important that the CPM tean 
continue this dialogue with management in order +o be aware 
of future developments that may have impact on the immediate 
or future performance of the computer systen. A well 
informed management, especially in the area of how well the 
computer system is performing, will be in a much better 
position to understand the computer facilities problems and 
needs, and will be more apt to take a positive role in 


procuring recommended resources or change. An awareness of 
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Management's near-term and future plans will most iixely 


infiuence the CPM teams recommendations to solve current 


performance problems oa sed upon current 8: 25ہ‎ 12205 
Procurement of equipment to provide adequate ps-zformancse in 
terns cf today's workload may not be sufficient te provide 


acceptable performance in the near future. 


1. User Requirements 


Users traditionally view the computer system from a 
very high level in terms of performance. They are generally 
interested in turnaround time or response time, accessi- 
Dility, reliability and availability of the system. If poor 
performance exists in these areas the user can become frus- 
trated and the "performance" of che user or user resources 
can be adversely effected. If 811 0 3 89, ex’st, this situs 
ation can cause conterproductive tension between the user 
and the computer system where the system includes the group 
of people who run and maintain the systen. 

& negative attitude can displace an enviornment of 
constructive comments with one of unconstructive fault 
memdang Criticisms that can easily turn into a cynical view 
of the computer system by the users. If allowed to 
continue, this situation can evolve into a situation of 
conscious or unconscious acts of sabatoge on both the users 
part and on the computer management group. The obvious net 
result in this type of situation LS a gross reduction in 
overall system performance produced predominately by 
external forces (user attitude) that may have been influ- 


enced by some internal ones (slow response time). 
a. Timeliness 


Few users actually keep track of the exact times 
for execution of their jobs, but rather develop an intuitive 
feel for these times. The CPM team should therefore obtain 
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these times from system accounting log files. This SEES 
of information about batch jobs works quite well. However, 
for interactive jobs that measure performance through 
response time other methods must used . The development or 
remote terminal emulators (RTE's) and intelligent hardware 
monitors has made it possible to accurately measure interac- 
tive response time of a large number of terminais. Tha RTE 
concept uses a separate computer from the one which is being 
measured to serve asa terminal activity generator (RTE). 


pa 


-— de 


The RTE generates activity that looks to the system being 
measured as if it was the activity of some 'n' terminals. 
"Usually, the programs in the RTE are parameter-driven and 
may be altered overa wide range of emulated activity", 
[Ref. 1]. One very simple method used to obtain a relative 
measure for response time is to keep track of the responss 
time of a mix of programs, that remain unaltered or 'fixad' 
and are executed at the same tines throughout a workload 
period (hour, day, week, 2tc.). Tha mix should include jobs 
that are I/O intensive and CPU intensive. From the CPM 
EME OE view, RTE-like monitors can provide an accurate 
record of users interactions, which reveal exactly what 
lev2l of service each user is actually receiving. 

Jobs should be given a priority and classified 
according to their use of resources. For example, +o take 
advantage of CPU and I/0 overlap, experiments shoulä be 
conducted to find which jobs in the same priority class are 
CPU intensive and which jobs are I/2 intensive. In order to 
achieve more effective performance of the system jobs with 
CPU intense operations should be scheduled with jobs having 
intensive I/O operations. Obviously, two jobs competing for 
the same resource (either I/O or the CPU) will experience 
longer turnaround time. 

Since performance can be severely reduced, it 
can be counter productive to schedule debug and development 


time ducing times of intense production runs. 
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Timeliness can be a function of output medium 


and computer system administration policy. This comes down 
to placing a value cn resources: user resource (tina) 
Versus system resources (printer paper, cpu time). For 


example, at The Naval Post Graduate School, upper and lower 
case printer output for students for a specific job is only 
avaliable once every twenty-four hours. Thus, the effective 


turnaround *ime for jobs requiring upper and lewer cass text 


is twenty-four hours! This becomes very interesting at «he 
end of an academic session, especially when graduating 
Seuaenes are trying to finish thes3s. Ie e possibla zii- 


adequate student training aimed at reducing the amount of 
printer output, could increase the system's overall effec- 
tive performance. 
Response and turnaround time is also a function 
of communication mediun. Modems ars a good example. À 
modem capable of 300 baud is noticiably slower than one 
enening at 1200 baud. If it takes fifteen minutes to 
transfer a file on a 1200 baud modem, it will take one nour 
to transfer that same file on a 30) baud modem. This is 2 
four hundred percent decrease in ovarall system performance. 
In this case, the user could find something élse to do while 
waiting for the transfer co complete. However, in an intear- 
active situation taking 15 minutes to complete using a 1200 
baud modem could take as long as one hour. This is also 
true with networks and the communication medium when they 
must use public communication lines to communicate form node 
to node. Thus, the performance of a node requiring service 
from the network can be greatly influenced by the communica- 
tions 110K, an external resource to the node's computer 
systen. 

Response time of a user can effect the perform- 
ance of the system since the response of the computer is 


dependent upon the user giving some command or answering a 
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AUS y Ta an interactive situation. The responss -ine of a 


user who must depress several keys to enter 2 string of 
alphanumeric characters to enter a command or respond to 2 
quer will bə much greater than a user who must sinply 


depress a function key, uses a light pen, or a touch panel. 

This iS especially true and most frustrating to a user whe 
has little cr no experience at a keybord. For highly inter- 
active sessions, the "mouse" (an input device tha? resides 
on any flat surface next to the terminal sc-zeen and whose 
movement about the flat surface is -z3fected in the movement 
of the cursor on the terminal screen) provides a much nore 
desirable human-machine interface resulting in impozved user 

espose time and overall increased performance. The user of 
a ncuse Simply positions the cursor over the desired choice 
of response displayed on the screen and depresses a button 
located on the mouse. 


B Accessibility 


Basically this involves being able to convi- 
nently get at all ccmputer system resources necessary to 
complete a desired task. If a person, who constantly uses 
the conputer systen, nust leave their office area to gain 
access to a terminal, then the terminal is not considered 
very accessible. These resources include copies of systen 
and resource documentation (to include terminal sign on/off 
proceedures, printer/plotter operating instructions, etc.) 
as well as hardware resources. 

System resource demand and requirement statis” 
tics can be obtained from system accounting log files. Fron 
this information the most effective geographical placement 
ef resources can be determined that will provide the 
greatest benefit to the users as a group. 


32 





Cone. aod 7۶7 


1 


Frequent system crashes or down time can be a 
nuisance, especially <o the interactive users. 23926 0 TOT 
Should be recorded on mean- time-between-failure (4TBTF and 
mean-time-to-repair  (MTTR). A high frequency cf system 
failures could indicate a weak hardware or software link or 
bug in ¿he system. Problems in this area are very hard to 
solve due to inadequate information. Again, accounting log 
files could help unccver a pattern, possibly related to 2 
Paeemcitat job or ccabination of jobs running concurrently 
that could te related to the systsn problenm. More discus- 


sion on this subject is in the section on Vendor relations. 
d. Availability 


In a very broad sense availability is the 
percent of the total time the system can directly serve the 
users to help meet the goals and objectives of the organi- 
zation. Scheduling, system maintenance, system failures and 
repairs influence system avaiiability. An extreme example 
dealing with both availability andi accessibility concerns 
the sclution of one computer facility to a problem of insuf- 
ficient terminal availability. There were simply not enough 
terminals for the demand of users who needed then. The 
"Solution" devised by the system administrator was to allow 
uers to log on only once each day. Apy attenpt to log on 
the system again, in the same day by the same user, wculd be 
denied. This "soluticn" caused people to "hang on" to their 
terminal by running endless jobs while they were not actu- 
ally using the terminal, so that they could have it avail- 
able, if needed later in the day. A better solution may 
have been to extend the shift and schedule users for 


specific terminals. 
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72٦62 EUNCCIOLS such as training Seminars, uses 
consultation, system documentation and the need for regular 
computer center news bulletins are difficult requirements +o 
determine and evaluate. 

Once again, the availability of detailed accounting 
data can be of use in this area. Frequency or infrequency 
of use by users of system software tools and. systen 
resources in general, as well as fraquency of compiles can 
help determine which users may require initial training or 
additional *raining in the use of system resources. A new 
programmer with a very optimistic outlook always included 
ee Catalog job control statement after the compile s«ate- 
ment in all software under development. Consequently, the 
operating system would catalog a job even though the 
compiler encoutered fatal errors ani the compile was unsuc- 
cessful. The programmer was  needlessly wasting system 
resources ard contributing to inefficient: use of the 
computer. Apparently, the progranmer was unaware that the 
operating system did provide for a conditional job control 
statement which would have prevented the catalog if the 
compile were unsuccessful. Ory possibly the programmer 
added or deleted some code to an existing program and failed 
to take out the catalog statement until assured that the 
compile was successful. If a new software tool is infre- 
quently used, a training session may be required. 

It is also important to aaintain a well run and 
responsive user problem report or suggestion/comment system. 
Users are a very good source (sometimes the only source) for 
finding system hardware and software problems or bugs. 
Users can be encouraged to participate in the system by 
always providing quick and inmediate sincere feedsack, if 
only to reveal the status of their report or comment, and 
thank then. 
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SEE Resource Utilization Report 


amp cum Sa Í Fe- ë n å ےک یت‎ 


Many Computer facilities must use accourting data 
for user charge back systems which are necessary =o obtain 
the funds required to operate ani maintain the computer 
facility. These reports are usually very high level, gener- 
ally providing totals for categories of resource usage, but 
the system uses more detailed information to generate these 
totals. This detailed information about user resource 
utilization should be made into a User Resource Utilization 
Report and given to each user showing by job, what rescurce 
is being used and at what amount. The user shculd be 
adequately trained tc take advantage of this information. 
Smee accountability for computer resource utilization is 
estatiished and made known to the users, they are frequently 
encouraged to cut down on wasteful use o£ computer 
resources. This can be especially true in the case when one 
user who needs a resource or more of a resource, knows what 
user or users are possibly wasting that z2source or using it 
205277616 77۰+ The net effect can be a reduction of user 
demand for critical resources and an increase in system 
perforıance. 

It should be mentioned that accounting packages span 
the range of levels of detailed information they collect 
about jobs in the system and about the systen. 
Traditionally, larger, nore expensive systems, like IBM, 
have accounting systems whose log files contained more 
detailed information about individual jobs (number of I/O 
requests, amount of memory used, number of pages of printed 
output, etc.) and the system; whereas smaller, less expen- 
sive systems have accounting data 5f less detail. In few 
eases is it found that the accounting log files contain all 
the information required, and supplemental software is 


required to gather this additional information. Reviewing 
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exes Source code listing ard “he documentation of an 
Becoumting pregram can proivide a Jzeat deal of infornz 
ME What, where and now to gather data on job and system 
performance. 


C. RESOURCE MANAGEMENT 


The CPM group shculd be able to easily monitor which 
resources ars underutilized (for example cardreaäazs, 7 
track tape units, etc.) and which resourcas ara having 
Barfieulyey geetong the workload demands. Using this infor- 
eon, obtainable through the accounting oackage and mnedi- 
facations to it, the CPM group will be able tom 
report,  reguarding elimination of under utilized resources 
(shifts, equipment, etc.) or the procurement of additional 
resources (memory, disk drives, tape drives, etc.). Through 
pussescomputer psrforsance evaluation efforts, direct cost 
Savings as well as improved performanc2 of th2 svstem can be 
raalized by the computer facility. 

BrnsoUmation collected in a historical database in “his 
area can bean invaluable aid ín predicting the near and 
Meno) term pattern cf growth  expeczed for the computer 
pacilitw. "Number of jobs compisted per month, percent 
utilization of major system resources, hours of systen 
availability ars several measures applicable EO ERIS 
problen. Performance data is therefore valuable not only 
for enhancing the present system, but also for constructing 


models of future resource requirements". [Ref. 4] 


dis ommunicating With Upp 


Tr ne u aS a Lm x 


Management 


8 


It should be a prime responsibility of the computer 
facility to report tc upper management on the status of how 
well the computer facility is meeting it's part rin 
supporting the organizations objectives and goals. Computer 





system performance is a key issue in this commitment and «hs 
DaMOSrOUuD can provide much of the required infozmatios and 
recomendaticns in this arsa. Reports should be utiílized <0 
convey this information to both the computer facility ard to 
upper management. In addition to these reports the computer 
facility manager may require additional information abcut 
the performance of the system at a more detailed level. 

The form and format of these reports can vary 
greatly duse to sach organizations reporting requirements, 
but some general guidance can b2 given (Ref. 1], [ Ref. 4) : 


e Status reports shculd be regular, concise, and prefer- 


ably graphical in nature. 


» The amount of information reported should not exceed 
management requirements. TOC oe TOO Ofton" is a 


probism common to many performance reporting schemes. 


e Information should be at a level of abstracticn which 
upper management can easily digest and understand, but 


Su@eevetent to support the decision making process. 


M ussreports snouid compare the center's current level of 


perfcrmance against a set of predefined goals. 


Most computer facilities interface regularly with 
vendor marketing and technical support personnel. A positive 
relationship with technical personnel can be invaluable 
especially in the areas of hardware and software reliability 
and maintenance. A joint review of hardware and software 
performance data on a regular scheduled basis by the 
computer facility personnel and vendor representatives will 
help foster mutual respec and understanding. Tc 9۹ 


system problems such as the frequency of tape and disk 
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lC GI-OZS, the cause and duration of system czashes, an 


as 


variations _n other system performance measures will orcviá= 
CII Manager with a folder of factual information to 
document the occurrence of these problems with which the 
vendor must deal. The capability to identify the source as 
well as the existence of errors becomes especially important 
in the multi-vendor computer system enviornment, in whieh 
each vendor points at the other vendor as being the nain 
source of the prcblem. 

Any proposed hardware and/or software modification 
will most certainly affect the parformance of the system. 
They must tnerefcr2 be evaluated from the total system point 
of view, not froh just ore aspect, as to: 

e Utility 


e Cost Effectiveness 
e Impact cn total System Performance 


"A modificatior to a system scheduler which is intended to 
increase batch throughput but which inadvertently degrades 
interactive response time fails to consider total system 
performance. Measurement tools and techniques can be used 
not only to detect performance problems, but also to antici- 


pate them and to prevent their occurrence." (Ref. 4] 


D. INSTITUTING THE CPM PROGRAM 


Measurement and evaluation techniques are available to 
support the efficient and effective management of a computer 
faced d ty. How to introduce them into *he computer system 
enviornment can be a rroblem. Some questions from [Bef. 4], 
properly asked are: 
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e How often should the information obtained from p2rforn- 
ance data be reported to the computer facility adminis- 
puogtor:? 


e In what form should it be reported? 


۶۰ ls the role of the computar facility adminis-rator 


in instituting CPM procedures? 
1. CPM Reporting 


Figure 3.1 (Ref. 4], shows typical computsr systen 
life-cycle, ptogressing from an analysis of réquire2ments to 
the final installaticn, operation, and enhancement of the 
selected system. In each phase o£ the computer life-cycle 
performance measurement and evaluation should provide a base 
for satisfying the informational needs of the computer 
bxchnzftv— adninistratcr. Performance data is as useful 
during the requirements analysis phase as it is during tas 
system enhancement phase. Every installation, reguardless 
of size, should incorporate a system performance reporting 
strategy during each phase of its system's life-cycle. 

Avallability alone should not be used to determine 
the types of data to be collected and reported. Instead, 
the data should first be selected as to the informational 
requirements of the computer facility, which are determined 
by the computer facility administrator's scope of responsi- 
Belity, Each report should provide a historical trend of 
mie fLaciiaty’s performance. Updates should be performed on 
a regular basis (depending upon the nature and importance of 
the information), and should contain specified performance 
eziberla. Control limits should be established from this 
criteria and placed on the performance charts. Control 
limits are a values chosen to represent the boundaries of 
acceptable performance for a given system variables. These 


variables and their associated control limits could be 
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Figure 3.1 The Computer System Life Cycle. 


40 





posetive-directed and indicate ths facility's abilicy < 


O 


meet certain specified objectives such as one-hour averagz 
batch turnaround time and 3 second terminal r=sponse time 
Others are process-directed indicating tha level of pezform- 


ance of internal system rescurces like the CPU, disks, and 


memor y. Exception reports should be generated whenever 
control limits are exceeded. When appropriate, an in-der*th 
study may be recommended +o determine tne specific cause (s) 


Bor tne exception and appropriate solution for its correc- 
pron. Reports should be simple and consistent with the 
NES: facility's responsibility to it's users in +h= 
areas of turnaround time, reliability, and userz suppcrt; see 
dure 3.2 for a turnaround time report, figure 3 er 
BEUOEDISty report, and figure 3.4 for user support rspor* 
[Ref. 3]. 


Determining ccntrzol limit values is dependent upon 
configuration and capabilities o£ sach individual computer 
luv as well as on the goals of the organization. Past 
performance may be used with caution as a standard agains“ 
which current performance is evaluated. Past performance 
does not necessarily indicat? good or poor performance, 
although it is a reliable indicator of the baseline systen's 
natural reaction to various workload demands. 

The perfomance reports gensrated by the CPM group 
should always be made availabl2 tə and discussed with the 
user community. Publication of "performance charts" in the 
facility's newsletter or on the bulletin board is an excel- 


lent vehicle for acccmplishing this. 


The computer facility administrator (management) 


should play a major rcle in the implementation and long tern 
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Turnaround Time Report 


Turnaround Time Report. 
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Freauency: Daily 
Performance Criteria: 


l. The average length of system 
interruptions shall be less 
than X minutes. 

2. No interruption shall last 
longer than Y minutes. 

3. There shall be no more than 
Z interruptions per day. 
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Figure 3.3 System Reliability Report. 
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User Support Report 
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Performance Criterion: 
l. The number of inquiries to the user 


support group shall not exceed X 
per week. 
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Figure 3.4 User Support Report. 
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Management of a CPM progran. P E 0 
۰۰۰۰۰٦ 7‫ CPM Management should ba pertormed to insure -hat 
2 E 


the basic qcals of the CPM effort az 
a. Define Scope and Objectives 


AS menticned eariier, the CPM management should 
develop and have a clear understanding and definiticr of the 
Scope orf its responsibilities. Next, the proposed CPM 
program objectives should be established and the information 
requirements should be defined in order to insure that only 
pertinent data will be collected during «he later phases of 
eae CPM program. Era tae —navgse Of the actual reports 
to be produced, the frequency wit hich chey shoula be 
together, and the performance criteria related +c each 
should be determined. CTO TS IDDIA EO mention aga2n; 
tna: tbe reports should be clearly presented and contain 
Soy @Heacningrul and pertinent infornation for the readers. 


b. Determine Approach 


The critical Seesourecs™ ter a successful “CPM 
program is net modern, up-to-date measurement equipment, bux 
skilled, knowledgable parsonnel who are intimately familiar 
with the computer resources being measured and the tocls 
being used, and who have the analytical ability to appraise 
and interpret th2 measurement data. AS mentioned earlier, 
without such a resource and a well conceived CPM program, 
the return on the investment in tine will most likely be 

isappointing if not disastrous. Two possibl2 sources exist 
for these key people: current (or future) employees, or 


Consultants from outside firms. 


The former of the two sources is preferred since 
it allows for better project control, and has the additional 
advantage that the in-house personnal are more aware of the 
objectives and internal workings of the organization. 
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Some advantages or the outside consultant 
Brraccehrare its cest-effzctiveness (for crganizations u 
no available performance measurement resources--l.<., 
personnel, monitors, etc.), its objectivity, and the oppor- 
tunity for knowledge transfer from the consultants to 
in-house contacts. Possible disadvantages ci contract 
But include the consultants! lack of familiarity with the 
internal operations of the organization, and the possible 
friction that may result between the consultants and 


in-hous? versonnel. 
c. Control and Review 


"The failure of many performance improvement 
efforts has been traced to the lack of genuine management 
تد وٍ9‎ >>> 308 support" [Ref. 4]. A probable cause of this is 
having a CPM group that is not totally dedicated tc the CPM 
EMO hat CEM is not their prime responsibility. They 
Become Spread “too thin” and if CPM has a low priority, it 
suffers the consequences. Again, «his problem can be solved 
by making sure that at least one member in the CPM mamage- 
ment group has CPM as atop priority and this member can 
od Cate full time te the CPM progranm. Another possible 
cause in the decline in interest in the CPM program can 
result from false hopes and unraalistic expectations for 
cost savings, especially if major expensive measurement 
resources like hardware monitors aust be purchased. The 
initial costs of a major CPM program can be well over 
$100,000 in hardware resources alone. 

The total CPM program should be reviewed period- 
ically to insure changes in informational need are reflected 
in new CEM reports. Existing reports should be examined to 
determine their current relevancy; irrelevant reports are 
cften generated when the need for them has long since 


passed. 
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Many of the resources required for CPM have been 
mentioned alorg with their cost. Perhaps the most signifi- 
cant costs of any CEM program are those experienced during 
the start-up phase of the effort. Such costs usually 
include: the program's initial planning, system modifica- 
tions to acquire the necessary data, acquisition of comaer- 
cial products and packages, and the development of report 
generation mecharisms. Once underway, continuous rsportzng 
demands other costs and resources: System overhead +0 
conocer fogaance data, processing time to analyze the 
data, and manpower tc interpret the data and maintain the 
reporting system. In addition, in-iépth studies prompted by 
exception reports and an intolerable decrease in performance 
require the use of rather sophisticated o0ls--notahbly, 
hardware and software monitors and modeling programs. The 
cost of these tools varies with the type and accuracy of 
data to ke obtained. The additional costs of personnel anc 


training may, once again, make it more cost-effective + 


O 


hire outside consuitants. 
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IV. CPE TOOLS AND 


4t3 


IHE SYSTEM LIFE CYCLE 

As mentioned earlier, the two most important and effec- 
tive tocls for evaluation are visual inspection and common 
sense. The use of other tools without these will mest 
likely be ineffectual. The additional CPE tools avaliabl> 
anclude: 


e Accounting Data Programs 
e Procram Optimizers 

SSE ace Monitors 

e Hardware Monitors 

e Benchmarks 

e Modeling 


CERCOS generally fall into ene of two broad catego- 
ries: measurement or predictive. Accounting packages, soft- 
ware and hardware mcnitors and source program optimizers 
fall into the measurement group. Modeling is a prediction 
tool and benchmarks are a point of reference for measure- 
ments and don't really fit totally in one or the other of 
these categories. These tools and their use will be 
described as related to the life cycle of a computer systen. 


A. CPE TOOLS 


1. Accounting Data Reduction Programs 


E M ern => -> En Se aa 


The need for an accounting program grew out of the 
requirement to fairly distribute the cost of the services 
rovided by a computer system amongst it's users. Today, 


mos: systems come wir accounting programs having varying 








Huscees ot capabilJty to gather information about *he system 
and individual programs. In early multiprogramming envizon- 
ments, it was noticed that essentially identical runs coulà 
generate a wide range of charges. This caused managers and 
users to search the details of the accounting data collected 
and to develop ways to collect more detailed information in 
order to uncover the reasons for the charge variations 
uot. 1]. 

System accounting packages generally collect three 
nes of information: identity, Guanve vy, and  *ime. 
Identity da-a includes such things as program name, charge 
sde”, device number and type, and other information that is 
useful for categorizing accounting data like programmer name 
AMS er neme, type of run (test or production), priority of 
mug, etc. Quantity data involves the amoun*t of some-hing 
used by a resource such as the amount of prime memory 
requested or used, total secondary storage space used, count 

£ I/O requests to a disk or tape, number of pages printed, 

EC. This data can be at a very detailed level, i.e. the 
actual rumber: of logical records and both the block count 
and the number of records ina block. Time data can show 
the amount of time used by a job or user for the different 
resources of the system. 

Àccounting packages produce reports which show 
processor time use, total memory requirements of the work- 
load, and give all the statistics collected in a summary 
fermat for review. Modifications to the accounting package 
programs and the development of new programs can provide 
user/job utilization reports. 

Accounting data can point to those programs that use 
the most system time and resources. These programs are 
prime candidates for increased system performance through 
improved program code and resource utilization. Morris 
(Ref. 1], gave examples cf this in the chapter on accounting 
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data. in one example, it was discovered that a disk f 
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unblocksd card image records. When the records were block 
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to track buffer slzse, the program erecuted twice as fast a 


LJ 
ern 


used considerabiy less secondary storag=. Another exanrle 
involves a program using multiple files zhat were stored on 
the same disk. By placing certain files on differert disks, 
device and channel contention was reduced which resulted an 
increased sffective utilization of the system resources. 
This is a case where the hardware nay have been performing 
efficiently but the system resources were not being used 
effectively. 

Accounting data can also be used for effective job 
scheduling. Tarougihastailed studies "OI resourc sa 
major jobs a nore optimum job schedule 'mix' can be achieved 
that can increase system performance, especially i 
active nultiprogramming enviornment. 

Accounting packages ars “he most widely used CDE 
tool threughout the industry. Development of more compre- 
hensive accounting programs has lead to more sophisticated 
and diversified applications of the accounting data in +erns 
of system performance evaluation. Morris (Ref. 1j, says 
that "One of the most promising developments of the CPE 
field that has largely been made practical by the avali- 
ability of acconting data and suitable reduction packages is 
Kolences' Theory of Software Physics" [Ref. 10]. Àccording 
to Morris, this work provides a completely rigorous defini- 
tion of software "work" and "powar" in a modern computing 
equipment context and applies the physics-like terms and 
definitions to practical applications in 8313 9 sand 
charging for system usage as well as in the more encon- 
passing field of computer system capacity managenent. The 
text 'by Kolence is suited to individuais with an interest in 


mee full potential of accounting data and 15 one possible 
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theoretical basis for an all-encompassing description of 


computer system performance. 


Software monitors are simular to accounting programs 
but collect much more detailed information. They can make 
measurements at the step-by-step instructicn «execution 
level. Software monitors are programs that are attached to 
the operating system and capture event and timing data 
directly ízcm internal tables setup and maintained b 
operating system. Data Sample rate can be varisa b 
user and captured data is stored on tape or in disk files. 

Software monitors are operating  sys-em/hardware 
dependent. Even a mcdest operating system change can impact 


a software monitor program. Software monitors usually fall 
into cn2 of three groups: Sampling, time-driven or event- 
dei ven. 


Sampling monitors EN sTact an sanplel OF subset of 
the entire populaticn of interest and through the use of 
Statistical methods inferences ars made about the entire 
Population. Variation of the sample size provides a range 
ef confidence intervais to suit the accuracy requirement for 
a given measurement. Event-driven software monitors rely 
Hpome™ nooks’ which are inserted into the code to be mori- 
tored or by a detection of a change of state. A hook is 
Simply an embeded instruction that triggers the collection 
Cc predetermined data by the software monitor. Data collec- 
ron camsalso be triggered upon tha completion of "n" occu- 
rences of a hook if needed in order to prevent excessive 
monitor overhead or "artifact" from dominating the program 
being measured. A change of state is defined as the tran- 
Sise@ome from one activity to another activity. An activity 
can be a change from privledged to unprivledged use of the 
CPU, the change from CPU wait to CPU busy, etc. Changes of 
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ER es generally occur less frequently then do hooks ani 
tHerefore performance data ús usually collected a-  sact 
Gecurence. Tame-ariven daza ccllection uses a clock or 
timer and at fixed intervals of tine collects measurement 
data by interrupting the execution of the program under 
measurement. The most capable software monitors use a 
combinaticn of these three types of data collection and 
selection methods. Software monitors use system resources 
i.e. CPU cycles, memory, etc.) that must be shared with ths 
program cr programs that are the subject of the evaluation 
effort and can therefore "contaminate" the analysis by its 
very existence. 


3. Program Optimizers 


aug {e ae S| å 


This tool is a subset of both accounting packages 
and software mOnators. Actually, the program optimizer does 
Poewaditeect ly potimize code, but rather it points out ے83‎ 
IE the progra that would produce the most benefit fron 
programmer provided optimization. Since program optimizers 
are compiled together with the program of interest, they are 
computer dependent. Program optimizers or "analyzers" can 
determine those parts of a program with the most activity. 
Meese are called the “critical part" of the program and 
typically "comprise from 1% to 10% of all program statements 
mma job or program. [Bef. 11] Unfortunately, progran 
optimizers are commercially available for only two major 
high-level languages, FORTRAN and COBOL (Ref. 1]. 

Application software is not the only software of 
interest when considering factors that influence systen 
performance. System software (operating system, utilities, 
etc.) also use and ccnsume valuable resources and should not 
be excluded from analysis consideration. Control programs 
analyzers (CPA) help to analyze this class of software. In 
sone operating system development, the only software 
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auzdelines are tha* it works. HOW efsiciently a 
tively it works aze cf secondary impoztance. 
Eeerlon Wili discuss in detail ths improvement of syste 
performance through improved softwares practices in the crit- 
Bear section of a pzogran. 


4. ar dwar 


KO 
= 


Qnitors 


Hardware monitors are external electronic devices 
that ars attached though the use of probes, to the various 
internal connections of a computaz system to sense and 
record changes of state in the components. An intelligent 
hardware monitors not only have ‘the capability to probe «he 
hardware bu: can also be connected by a line of communica- 
tion to the measured computer system. The intelligent 
monitor incorporates its own computer, complete with moni- 
Boing Software and uses “his line of communiction to actu- 
ERU control ae EC heni-orz.g process via coordinated 
communication between the monitor and the measured computer 
System. 

Although hardware monitors are becoming "easier" to 
use, they are still the most demanding in the area cf tech- 
nical knewledge required, especially in the area of system 
architecture and implementation down to the circuit level 
and require the largest financial investment. In addition, 
hardware monitors can be easily misused or incorrectly used 
and can generate data at such a dataiied level and in such 
large amounts that inexperienced users often find themselves 
overwhelmed with data after only a short period of data 
collection. However, their low level of detailed informa- 
tion gathering capability,  miniminal operating overhead and 
the fact that they are generally system independen*, makes 
“hem an attractive and desirable monitoring tool. The 
choice of selecting between a hardware or software monitor 
will te discussed in greater detail in a later chapter. 
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5. Benchmarks 


Benchmarks are programs that ace designed to spscir- 


eony represent the workload or the se* of workloads cha 


re | 


E. or are anticipated to exist on current or proposed 
future computer systems. Benchmarks act as a reference urit 
of measure *o be compared with the results produced by other 
CPS tools. 

A small set of benchmarks that have been validated 
as accurately representing the total workload at a given 
period in the history of the syst2m can be extremely 
valuable to a CPM program. Performance data produced by 
periodic executions cf benchmarks allow the CPM group to 
evaluate changes made to the computer system. This periodic 
comparision process provides data to easily chart the posi- 
tive (improved performanca) and negative (degraded perform- 
ance) impact made on the system. 


Modeling is a discipline in its own right and has 
been widely used thoughout many fields. Modeling makes use 
of mathematical descriptions to provide an analytic or simu- 
lated model of the system +o be studied. Modeling in the 
computer enviornment allows the detailed examination ofa 
conputer system or set of prograns before they actually 
exist. In this sense, models allow the "nature" of some- 
thing to exist without the acutal existence of that thing. 

The most important aspect of modeling is that it 
allows the question "what if. . .?" to be asked ina 
variety of different ways through the use of parameters, 
with the model producing results for each of these different 
requests. Modeling can also provide the results of applying 
a proposed solution as suggested through the use of other 
CPE tools, +9 a problem without actually implementing the 


\ 
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solution suqgestad. For exanpla, a model could try 


@errerent corZiguraticns of memory, charnels and disk uai* 


Ui 


Without ever installing she actual devices. This feature of 
modeling is extremely powerful, sxpecially in long range 
CPA/CPE efforts. Hcwever, a model must bé rigorously vali- 
dated and if possible verified with existing sytens. 
Developing a model is very time intensive and therefore very 
expensive. For a more detailed discussion of these tools 
see Morris (Ref. 1], in which a chapter (3-9) is devoted to 
ai scussing each tool. 


B. CEM AND COMPUTER SYSTEM LIFE CYCLE 


It is highly desirable to dévslop a CPM program «hat 
goes beyond the operation phase of the computer life cycle 
and encompasses the complete lif2 cycle from procurement 
phase to transition phase. Although not suitable for all 
computer facilities due to costs or other reasons, she 
following section lists some ideas to consider during the 


life cycle of a computer systen. 


This phase of the life cycles can be the most impor- 
tant, in that the framework for the potential expansion of 
the computer system is decided. This decision obvicusly has 
the greatest impact on all the fcllowing phases. Ie iz at 
this phase that the mcdeling tool is most powerful and most 
enlightening. After arriving at a workload specification, 
modeling can be used to allow alternative parameters +o be 
evaluated for such resources and components as the type of 
peripheral device best suited to a particular file structure 
or configuration of memory that will be needed. Ranges of 
relative speeds and capacities can be computed from the 


model that will allow selection to begin on major equipment 
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suen aS CPU, memory and supporting 2quipment such as «ape 


drives, disks and printers. Lis, the Understanding coz 


D 


proposed initial and fu-ure anticipated workload and =quip- 
ment will aid in the selection and developmen* of benchmarks 
of the system. Benchmarks will play a major role in 
preparing the requests for proposals (RFP), also known as 
procurement documentation, for the pending equipment 
selection. 

Vendors, once aware that a perspective customer has 
developed a model capability and has selected benchmarks for 
a System, are more apt to become coapetitive by making modi- 
fications and custom tailoring their proposal to meet the 
customers needs more closely and at the best price. "The 
proposal review is ore of the most inpertant steps in «he 
system's life cycle and in the CPE effort because it is the 
one time when all potential suppliers are most intensely 
interested in the buyer's needs" [Raf. 1]. 

At a recent Computer Perfcrzmance Users Group meeting 
an example was given by a speaker, in which inadequate 
attention was given to matching the new system with the 
anticipated workload involved a government agency that 
purchased a computer costing less than $300,000. Having 
procured the new system, the agency proceeded to develop 
several million dollars of software to be run on it. The 
one system soon proved to be inadequate to accomodate the 
workload, and several more wers eventually purchased. 
Difficulties arose trying to interconnect these computers, 
and after the purchase of five identical systems and 
connecting them together the system still was unable to 
provide sufficient service to handle the workload at a 
satisfactory performance level. This situation would be 
much less likely to have occured if a CPM program had been 
in effect during the procurement phase. However, it is 


thought that many government organizations are forced to buy 
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inadequate computer power asa result of governmentation 
computer procurement procedures. Larger system pIocursnent 
require more paperwork and have longer procurement tinas. 
This may force some organizations to buy a smaller, less 
adequate system based upon the amount of procurement paper- 
work required instead of the ability of the system to handle 
the workload. 

The selection or final step in the procurement phase 
relies less on  perfcrmance factors since by this time only 
proposais that should be considered have provided positive 
benchmark results, and the ramaining selection criteria is 
price, delivery schedule, vendor support, conversion costs, 
etc. 


Zwei seal lation Phasa 


Due two Wos® important conzserns in this phase are: 
establish the system configuration that is best tailored to 
the workload, and verify that the delivered and installed 
equipment does indeed perform as par written specification. 
The former concern means going ov2r any and all workload 
characteristics again and verifying that the equipment and 
zonemOuration is correct. This is the last chance to make 
sure everything is "right". At this phase the buyer has the 
upperhand in that the system is not "Signed off" yet and the 
vendor will be motivated to be most accomodating at. 
Modifications that can be simply made at this phase become 
increasing more difficult and expansive at later phases. 
The latter concern, that actual performance equals or 
exceeds advertised values, should be demonstrated by the 
vendor. 

It is at this phase that CPM/CPE personnel can learn 
a great deal about the new system directly from the vendor 
installation team. The buyer should get involved, tactfully 
"buy lunch", learn how to use the system diagnostics, and 
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generally learn everything that hey can from the people 
installing the system. If the relationship besween  *he 
buyer and vendor is positive and accepted by the vendor, 
fee more likely that the vendor technicians will point out 
bono strong and weak points in the system. The CPE 
personnel can gain extremely valuable information es a 
consequence of this relationship in that detailed technical 
information relating +0 performance evaluation such as 
system table addresses, pin location, probe points, etc. can 
be obtained from current as well as potential future 
Sone ac S « 


ge Is during the Operations phase that most people 
consider some sort of CPM or CPE effort. As has been shown, 
many potential problems reguarding initial and long tern 
performance can be avcided or solved prior to this phase 
rough tke use of a CFM program and CPE tools. Even wich 
ansehe pzior CPM efforts, a dynamic and expanding work- 
ioctl require continuous attention from the CPM group. 
Each new progzam should be reviewed as to it's impact on the 
workload and performance of the system. Accounting data 
along with supporting performance data gathering programs 
provide sufficient informatica to assess the impact. es 
problems are uncovered, more sophicated CPE tools may be 
required to pin point the appropriate solutions. 

If a model does exist that closely resembles the 
system, it may be able to determine the benefits and advan- 
tages of new products and options that become avalianle 
throughout the life of the system. Through the use of a 
responsive and cost effective CPM program and CPE efforts, 
the system's point of saturation can be substantially 
delayed. Thus putting off a considerable expenditure for a 


major upgrade or new systen. 





Once again, a system model is very useful and 
valuable in determining the capabili- 3-۰۰2 ۹ و9‎ 
system to handle an expanding workload and to reveal whet or 
where system changes and modifications will be required in 


order to satisfactory maintain this workload. The limit o 


th 


this process is system saturation, where additional enhance- 
ments to the system are not possible or not practical or not 
cost effective. It is at this time that the computer systen 
E move into thel transition phase oí it's liifs cycle. 


iSi +t 


و 


Phas2 


Be this point, the CPM group should have gained a 
great deal of experience, become expert with the dynamics of 
the organization's workload growth characteristics and 
expert on asystem that has managed to adequately process 
the workload until this time. Along with an indepth 
personal knowledge of the installation's performance 
Iur the CPN group shculd be well prepared to aid in 
system transition once it has been predicted. 

An organizaticn that has effected and penefited by a 
CPM philsophy and program should bs in a very good pcstion 
to make an orderly and intelligent transition in which 
major, unanticipated performance problems will bes minimized, 
Eu Her coecur at all. 

The above discussion of CPZ tools provided some very 
general guidelines in the use of these tools during the four 
phases of the life cycle of a computer system. However, two 
CPE tools that seem to be used most often throughout the 
Hre ceycle CEM éifort are accounting information and system 
and workload modeling. Due to the low cost of accounting 
packages it may be desirable to implement the most extensive 
and detailed one avaliable. It should be an ongoing effort 
to beccme iacreasingly knowledgable about the operating 


system software so that accounting programs can be 
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Supplemented with additional capability in order to supply 
all the information that is desired in the CPM process. The 
source code of the accounting system is an invaluable docu- 
nent in this regard, especially in answering the "where" and 
"how" questions in obtaining the data. 

As useful as models seem tə be, they are extremely 
hard to develop and validate to any degree of accuracy in 
imitating the actual situation. However, it is suggested 
that, as 3a minimum, a resource or ccmponent diagram ba 
constructed of the system showing capacity, speed, buffer 
size, transfer rate, etc. of each component, as well as 
lines of communications and/or channels and their associated 
rates and capacities. Some  simpie caiculatior using these 
values can be very informative when trying to match devices 
and rescurces. The same idea could be followed for the 
workload where all the programs would be listed along with 
the resources and amcunt required as well as critical time 
values, such as response time. Even the Simplest of models 
can be guite helpful for example, when components (and asso- 
ciated speed, capacity, etc.) of a system are listed in a 
aroup as well as a description of will be running n the 
system. This "paper" model can provide a feel for potential 
problem areas such as mismatch of the number of channels for 
a certain number of devices, or that the amount of memory 
presently configuared will only allow a very limited multi- 
programming enviornment to exist. Table I shows «he primary 
and secondary tools required throughout the life cycle 
process as proposed by Morris (Ref. 1]. 


C. PRACTICAL ARCHITECTUAL ENHANCEMENTS 


Svcbodova (Ref. 12], suggests that the utility and 
Simplicity of the following architectual enhancements would 


make them practical tc implement. 
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TABLE I 
CPE Tools and System Life Cycle 


PROGRAM OPTIMIZERS 
HARDWARE MONTIORS 


CPE SOFTWARE MONITORS 
TOOLS ACCOUNTING DATA 
BENCHMARKS 
PHASE ACTIVITY MODELING 

Procurement Conceptual Design of Workload P 
Detailed Workload Specification P 
Equipment Specification P 
Request for Proposals (RFP) P 
Review of RFP Responses S 
Select Equipment Supplier 

Installation Tailor Workioad to Equipment SP SISIS 
Configuring Equipment SiP SiS 
installauon and Checkout E S 
Acceptance Tesung P SiS 


Operation Workload Implementation 
Program Reviews 
Configuration Enhancement 
Adding New Workload 
Projecung System Growth 
Predicting System Saturauon 


Y UM On 


Transıuon Review of New Systems 
Review of Potential Data Processing Needs 
Reuse Anaiysis of Owned Equipment 
Conceptual Design of Foreseeable 
Workload : 


YUU 


پ- 





P = Prımary Tool 
S = Supporung Tool 
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1. System Timer. The system ziner is a standard feature 
OL Many present computers. Accuracy of timeable nzas 
ures monitored by a software monitor is limited by 
the resolution of the system timer. The system «ime 
can be used in one or both of its modes: 

a) Data mode: The system timer provides time stamps 
tcr monitored events (times of day clock). 

b) Centrol mode: Interrupts generated when the +imer 
expires drive the sampling routine of a software 
monitor, or synchronize a hardware monitor with 
software activities of the measured System 
(upnterval timer). 

CO MTEC LON cotit. The instruction councert counts 
executed instructions, either otaliy or condition- 
aliy instructions executed in problem state, 
me uC ions executed in the partition,N, etc.). 

a) Data mode: The instruction counter facilitates 
measurement of parameters such as instruction 
SxecCecue2OnuLate, sins cuctzons executed per I/O, CPU 
3۷ 2 on. 

b) Control mode: Santi res time: interrupts, the 
1igstruction counter Can generate interrupts that 
are used for the purpose of program sampling. 

jJ. Memory cycle counter. The memory cycle counter counts 
memory references rather +han executed instructions. 
a) Data mode: The memory cycle counter facilitatss 

measurement of parameters such as memory rafererce 
rate, memory references per instruction, memory 
۰ت5‎ 112830 memory references between page 


faults. 


lThe term instruction is sometimes used for a register 
that holds the address of the asext instruction o be 
executed (program ccunter, “instruction address register). 
In this context, we are using the term instruction adáress 
register. 
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b) Control mode: The memory CVE COUN “Can 


Geverate —lmverrip ts that dzive a monitcr routins 


Sampling program for the purpose of axaninin 


4 


lecality of reference. 
Address match circuit. Many computers have a hardware 
debugging aid, the address match circuit. The 
address to be matched is loaded manually thourgh 
toggle switches. A match generates a pulse in the 
CPU circuitry, but this pulse is net recognized by 
Soeewane, —wulen ples a constraint on ths utility of 
such a feature as a monitoring aid. To becom2 a 
mcnitoring aid, the address match circuit has to be 
given interrupt capabilities. A Special. ة1‎ 
can be provided to load tha address match register 
under program control; the absolute address that is 
to be matched then does not have to be known explic- 
BER. A match pulse causes an interrupt that trans- 
fers Control toca mondtoring routine. The address 
match circuit thus provides a remote hook. Together 
with an associative memory of sufficient size, the 
address match circuit enables system software level 
instruaentation without any physical changes to the 
monitored software. Such instrumentation  couid be 
temporary (for testing software monitor routines), or 
permanent (if for some reason the monitored software 
must not be modified). In addition, match pulses are 


potential synchronizing Signals for a hardware 





monitor. 
Monitor call instruction. Memonitor call irserns- 
tion provides access to system monitor services. 


Such a feature has been implemented on the IBM S/370. 


The 'Monitor Cali! instruction (MC) operates simi- 
larly to the IBM Supervisor call (SVC). A special 
centrol register, called the monitor register, 
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meso asas a mask tor different moritoring Classas 
in the sane way a mask is used with system inter- 
CUBES A special  privilsged instruction, ! Sec 
Control Register! can change the mask. A 'Moritor 
Call‘ instruction then designates one of tha sixteen 
monitoring classes. If the designated class is 
unmasked, then execution of this instruction causes 
an interrupt that is procassad by the operating 
systen software. 

Hardware monitor plug interface. The function of 
this interface is to concentrate signals representing 
most frequently measured hardware events, and stabi- 
lize these signals for the period cf event duration. 
The interface is placed in a location that is easy to 
reach, with terminators that provide an easy connec- 


tion for a hardware monitor. 
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V. CHOOSING A MONITOR-HARDWARE OR SOFTWARE? 

If the decision has been made that the accounting data 
no longer provides the level of performance information 
required by the CPM group, then the nexr logical decision 
would be to choose between a hardwars monitor or a software 
monitor. The selection process reduces itself to picking 
the monitor whose characteristics most closely satisify or 
meet the user's requirements. A very informative article 
which addresses the mcnitor selection issue is "Performance 
Monitors-Hardware or Software?" [Refs 13}. The fcllowirg 
provides sone of the main ideas regarding the question of 
selecting a monitor. 


A. SOME GENERAL CONSIDERATIONS 


There exist at least three considerations common +o both 
+ - + > T ce 
types cf monitors: 
e Cost effectiveness -As a CPM program grows, the issue of 


how cost effective the next step is, should always be 


considered. Purchasing either monitor type will demand 
the wallocation of mote resources, both financial and 
personnel. Ae “MGULtOn wall incurs the following costs: 


the mcnitor itself, personnel (both -ime and training) and 
most likely the use of the existing computer resources for 
perfcrmance data, reduction and report generation and 
Special experiments. The initial cost of a monitor can 
range from several thousand dollars to well over one 
hundred thousand dcllars. All of these costs must be 
totaled and compared to the anticipated benefits fron 
taking this step. The net result will determine if it is 
cost effective. 
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e Definition of objectives -As wich the CPM program, a 
plan is needed for ever important CPE endeavor, espe- 
cialiy the purchase of a new product or resource. Issues 


involve: how the monitor will be used, what impact i+ wil 


ps 


have on the system, what will be measured, stc. 


e Personnel capabilities--This issue has been discussed 
earlier in this paper. The important consideration here 
is that the moniter will require properly trained and 
qualified personnel to use it and these personnel must be 


able tO Communicate with vendor support personnel. 


Weum@ucrace csi Stics of 83710832 and Softwares Monitor 








Table II presents a quick references for comparing 
C 


E 

and evaluating twelve the two types of 
monitors (Ref. 13]. An elaboration of each of these monitor 
tuac-sristics follows: 

1. Cost--Since a hardware monitor is a combination of 
hardware (some have full capability microcomputers) 
and a software package, Liv s ın walls cost can be 
usdserately, to considerably more than that of a soft- 
ware monitor. Operating costs of a software monitor 
are considerably lower than those of a hardware 
Wemecto>, since its operation and configuration is 
relatively inflexible, and while set-up time fora 
hardware monitor can be quite extensive (finding and 
attaching prcbes, 26۶1 0 logic boards or 
Brogsagen, etc.) í: is minimal or not required (y the 
user) for a software monitor. Formal training in the 
use cf a hardware monitor as well as the design of 
experiments also adds to the cost of using a hardware 
monitor more sc for using a software monitor. 

2. System overhead--Hardware nonitors which only use 


probes add no additional overhead to the system being 
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monitored. Hardware monitors that have communica-ion 
links with the monitored system can impose some cver- 
head. software monitors can inflict substantial 
overhead on the system since they are programs which 
are run on the monitored system and require the use 
oi the system's resources such as CPU, primary and 
secondary storage and I/O sarvices. The severity of 
the overhead, also referrad to as "artifact", is 
dependent upon the rate at which sampling or meas- 
uring is being performed. With the exception of scme 
intelligent hardware monitors, Dern noni tors -rely 
upon the measured system's rescurces for data reduc- 
tion and report generation. Some real-time systems 
May not be able to tolezate sven the slightest amount 
of additional overhead imposed by a monitor, and 
would require a probe attached hardware moniter. 
Hardware/operating system dependent--With the 2xcep- 
-ion of some intelligen- harzdware monitors, hardware 
monitors are essentially independent of both the 
hardware and software of “the measured syste. n 
sone cases inadequate or inaccessable pin locations 
can be a very serious problem, preventing full use of 
a hardware monitor. Software monitors use system 
specific information and are therefore custom 
designsd for a particular hardware/software configu- 
raticn for a particular brand of conputer. Softwars 
monitors don't exist for all computers whereas, if 
performance related probe points exist on a system, a 
hardware monitor can be hooked up to i*. 
Precision--Precision in a hardware monitor is depen- 
dent upon the following monitor internal components: 
the resolution of the clock, speed of it's counters, 
accumulators and other internal circuitry. 


Therefore, it is desirable to use a hardware monitor 
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With a precision greater then that of the system 
being monitored. Since software monizorzs utiliz 
monitored systen's m eertalsclock Fandsche ability of 
the operating system maintained measursments, it's 
percision is usually lower then that of a hardware 
monitors potentical capability. Due to variation 
caused by the sampling technique typically used by 
SOTtware ponitors,  precison can drift. For the 
majority of measurements, either monitor will prove 
to be satisfactory. In those faw cases in which 
extreme precision must be used, hardware monitors are 
necessary. 

Capture of gualitative information--Software monitors 
can more easily explore qualitative information such 
as the contents of main memory, queue contents, 
program ID's, and other descriptive variables stored 
in tables. The ability of nost hardware monitors to 
explore qualitative values is restricted by the hard- 
ware connections. Some intelligent hardware monitors 
do permit some qualitative information to be trans- 
ferred to the hardware monitor via the use cf it's 
communication link. However, the ability of hardware 
mcnitors to collect qualitative information is 
limited relative to +hat of software monitors. 
Training--Hardware monitors, due to their very tech- 
nical nature, require more training which is critical 
to the degree of success attainable. Hardware moni- 
tors must be configured by the users which requires 
selection, location and varification of probe points, 
wiring of logic plugboards, etc. In as much as the 
hardware monitor is systen independent, Zee 
extremely  user/analyst dependent, and the better 
trained the user the more benefit can be gained from 


the monitor's capabilities. Software monitors 
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require relatively little training for implementation 
and operation. Software monitors have a "fixed" ser 
of Zeports and are more easily analyzed then 
reports from a hardware monitor. 
Flezxibility--Measure ment flexibility potential exists 
in both mWonitcrs. However, to achieve any degree of 
measurement flexibility in a software monitor,  modi- 
fications must be made to the code in the data 
collectíon progran, the data reduction program and 
the analysis and report generating programs. Shor? 
of performing these possibly extensible mcdifica- 
CIOnS, the software monitor remains relatively 
inflexible to change of the types of measurements. 
On the other hand, the hardware monitor is designed 
to be very flexible. In general, anything can be 
measured for which a probe point exists, although 
data reduction programs may be limited to only fairly 
standard measurements and reguire modification. 

Ease of use--Compared to hardware monitors, the soft- 
ware monitor is very easy to use. Activation and 
selection of desired reports is usually requested 
though the input of simpl2 console commands. 
Hardware monitors can take several days of set-up 
planning, coordination with the system users in the 
area of scheduling, and can lead to  disrup-ion, 
confusion and interference in the computer facility. 
Each new measurement which uses different  prcbe 
points requires reconfiguration and planning. 
Portability--There are two major divisions relating 
to portability: amongst different brand name systems 
and amongst same brand name systems. The hardware 
nonitor is extremely portable in either division. 
All the hardware monitor depends upon is the avali- 
ability of attachable probe podus. Software 


70 





10. 


11% 


12; 


mcnitors are cnly portable amongst systems “hat have 
the same hardware/software configuration. Since 
software monitors are relatively inexpensive conpazed 
tc hardware mcnitors, it may be as cost effec-ive to 
buy a few software monitors for dissimilar systems as 
it would be tc buy one expensive hardware monitor. 
One advantage of multiple software monitors over a 
Single hardware monitor is that the software monitors 


can run simultaneously and uninterrupted. A hardware 


monitor can only be active on one system and must be 
totally reconfigured when moved to another system. 
One other consideration is that hardware monitors are 


much more susceptable to "handling" problems in that 
they are delicate electron devices, and could be 
easily broken. 

Device monitoring--Hardware nonitors may be attached 
to any device having a connector. This type o£ 
attachment provides a differant "view" of the device 
then that possible through the use of a software 
monitor in that hardware monitors "stand at the door" 
to review data entering and leaving whereas a soft- 
ware monitor can walk in and freeiy explore the 
contents of the room. This is true in the case of a 
device such as a disk. 

Interrupt considerations--Software monitors can be 
adversely affected by interrupt conditions in the CPU 
in that they can be locked out, or they can take 
advantage of the interrupt capability to stop 
processing and allow the software monitor to explore 
any part of the system. Since most hardware monitors 
are external, they are not affected by intercupts. 
Expected life--Hardware monitors can be exected to 
remain relatively useful for a long period of time if 
connection pins remain avaliable. Software monitors 
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are dependent upon a specific configuation and minor 
.ت۰۰۰‎ tOo his Configuration Cad inpact the useful 


life expectency of a software monitor. 


B. MAKING THE DECISION 


As mentioned in the beginning of this section, the 


selection ofa monitor is primarily a question of n 


ua 


tmr 


a 
the specific needs of the CPM group with the capabilities o 


th 


each of the products avaliable. The preceeding twelv 


iD 


points should form the basis for making the comparison of 
what is avaliable. De importanee or inicielly detining 
objectives can not be over stressed. The selection process 
is more apt to be successful if it is known beforehand 
exactly what is to be measured, what questions are to be 
asked, and what desired end result is expected. In some 
instances, circumstances may exist that leave no choice but 
selection by default. A software monitor may not exist for 
a particular computer system, or a real-time or highly 
active multiprogramming system may not be able to tolerate 
additional overhead created by a software monitor, so a 
hardware monitor is the only choice by default. In unique 
applications, such as the space shuttle program, special 
measurements nay have to be made requiring the minimum cver- 
head of a hardware monitor. Sinca software monitors are by 
far the less expensive of the two, organizations that have 
limited budgets may have no choice othər than to buy a soft- 
ware monitor. Shortage, or nonexistence, of personnel poss- 
essing the required backround or general experience 
necessary to be trained in the use of hardware monitors may 
also lead to the same choice. The case studies found in 
Bookman [Ref. 14], "Saving from Performance Monitoring" 
(Ref. 15] and Sewald (Ref. 16] provide ways which other 
organizaticns have elected to choose a monitor and conduct 


méaSurement programs. 
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VI. SOME COMMON SENSE PERFORMANCE FALLACIES 


When faced with a problem, most people will either have 
or will soon have a "hunch" or an intuitive explanination cr 
hypothesis as to what is causing the problem, or what i+ may 
take to make the problem go away. In almost every case, 
extensive remedies should not be implemented until "proven" 
to be the correct or at least the best possible one. In the 
context cf computer performancz evaluation, the best method 
for proving a hypothesis correct is through careful measure- 
ment. If an intuition leads to the impiementation of 2 
remedy without adequate verification and proof, considerable 
time and money can be wasted. Therefore, one must be 
careful about the use of ccmmon sease guidance about wheres 
bottienecks might be in the system and what causes then, 
since it is very  sasy to be mislead in such a complex 
systen. 

This chapter will discuss some specific, possible, and 


* 


general fallacies cof intuition as noted by Carlson in hi 


0 


article "Fallacies cf Intuition in Performance Tuning" 
(Ref. 17]. 


A. SOME SPECIFIC FALLACIES 


1. Fallacy: During peak load processing the CPU busy 
increases significantly. 

This is a carry over from the old days when operations 
people would judge the behavior of the CPU by the activity 
of the noisy peripherals. Actually, the peak load 
processing can have very little impact on actual CPU cycles 
used, but can have significant impact on noisy peripherals. 
Additionally, in many cases i+ was found that the "CPU usage 
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Ec nebease only 24$" to 4% in CPU busy” (Ref. 17], 
however, a peripheral like the printer would become +wica as 
busy. An increase then in peripheral activity wou 
the impression that the whole syscem was much more active. 
Not surprisingly, and thou yerırieation, nany 
computer facilities wculd seek to increase the  capaci-y of 
their system. This normally resulted ina desire for nore 
CPU power or additional memory. IDviously in cases where 


the actual increase in CPU activity during peak workloads i 


in 


very small, modifications to the CPU would have little cr no 


(A 


|^ 


effect on the overall performance sf the system. What 
Of whe 


peripheral equipment based upon tha results of resource 


needed in this situation is a careful balancing 


utilization measurements, and a possible review (OMS 
accounting data to come up with a better job scheduling 
algorit im. 
2. Fallacy: The reason for slow terminal response times 
is slow disk access times. 

Some simple comparisions of only time and rate values 
for ccoperating devices and communications links could lead 
to the above conclusion. Terminals are attached to control- 
lers (which have fast internal speeds), that coordinate 
CPU's that operate in microseconds or nanoseconds, and disk 
units which operate in milliseconds. Therefore, it would 
seen intuitively sensible that the disks, which operate 1000 
times slower than CPU, are the most probable cause for 
delay. Without going any further than the comparisions of 
the above information, it seems that obtaining faster access 
secondary storage devices would improve response time. 

Carlson devised plan to measure those activities that 
influence response time. A hardware monitor was selected to 
record all major events cccurring during response time 
dela y. Factor analysis, a statistical technique, was used 


to indicate which variables might be influencing response 
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time the most. The results appear in table III and 


S 
Bat ask activity accounted for only 14% of the response 


LABLE III 
Response Time Components 





€ 
Activity During porcen ۴ | 
Response Time Response Tine i 
------2--22--2-2-2------ ~----- +---- --- - - - -- - - = -- 
rerminai Controller | 21% | 
Disks Only Busy j 14% | 
Channel busy | 53 | 
TP Superviscr Program | 15% | 
ES aaa pat | 18% | 
Non-interrupżable { 46% 
Logical OR cf all Items. | 943 | 
üncceounted for i 6% ۱ 
| | 

| 


tine delay. Replacing the old disk with one having cne-half 
the access tine (twice as fast) would only resuit ina 7% 
reduction in repsonse time. If ths access time were to go 
ہم‎ zero, the net result would be only 14% reduction in 
access time. In this operating system (IBM OS) overhead CPU 
functions (i.e. ron-interruptable activity) accounted for 
almost half, or 46% of the response time delay. 

This example frem Carlson (Ref. 17], points out that 
there are rarely one-to-one relationships between different 
devices in today's complex computer systems. The fact that 
a disk drive operates one thousand times slower than the CPU 
has only a small impact or minor relationship to overall 
system performance, The computar system is auch more 
conplicated than a one-to-one additive relationship. 


Therefore, in this case it was not so much the performance 
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of the equipment or what equipment was used, but rathar how 
mings were dane-i.e., the system software. A close lcok 
sew tuning of the interrupt and I/O handl¢rs could brin 
about much greater reductions in response time. 
oy hall lacy: Switching from single density to double 
density disks will cause system delay due to arm 
contention. 

Doubling the density while keeping the arm mechanism the 
Sane can be viewed as potentially doubling «he demaads on 
the access arn. In view of this, it is easy to see how the 
above intuitive belief could be supported. umher support 
for this fallacy stems from the fact that one would expect a 
fairly high number cf concurrent sseks to occur due to the 
highly active operating system that mus* Support a multipro- 
gramming environment and disk controllers that suppor 
Multiple disk units. The fact is, that going from single to 
double density has a negligible impact on overall perform- 
ance. Careful analysis of many operating systems indicate, 
however, that only rarely are concurrent seeks attempted on 
any of the packs on a given controller. Tke 9 
example was found in Carlson [Ref. 17]. A hardware monitor 
was used on disk drives, clustered eight toa controller. 
Pins were located that were thought to provide evidence as 
to whether or not a seek was in progress on any of the 
packs. The initial measurements sesmed to reveal that seeks 
were in progress 100% of the time, indicating that the disk 
controller was extremely busy. However, it was soon found 
out that the pins being probed were only indicating if power 
was on in the disk controller. This points out the very 
important need to verify that the pin does indeed provide 
the desired measurement information. One very simple, 
although not conclusive method to validate the pin, would 
have been to push the halt button on the system and notice a 
coca ponding halting of disk activity indicated in the 


collected data. 
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Mme Corret pinsWwer> located and verified and it was 
found that on a typical 8-pack disk subsystem, overlapped 
seeks account for less than 1% of the time. Ihe maximum 
seek overlap ever measured was only 3%. The experiment was 
conducted prior to and after the change from single to dual 
density packs. The system was stabilized with single 
density drives and measurements were made of CPU activity, 
channel activity, response time, etc. Then the most active 
disk pack was combined with the least active disk pack on 
one dual density drive. The same measurements were made 
again with no measurable difference in behavior. Even 
combining the ‘two most active packs on one doubie dersit: 
pack revealed no perceptable degradation or change in syste 


a 


activity as indicated by conductinJ the same test. Thus, 
careful measurements seem to dispute the intuitive idea of 
increased access arm contention as a result of switching 
from single density to double density disk packs- 
Therefore, the decision could bes made to go to double 
density packs based upon economic issues, in seat dual 
density disks may provide lower cost-per-bit secondary 
storage. 

4. Fallacy: Large core storage (LCS) or asynchronous 

memeory is always 2 good thing to put on a systen. 

There seems to be a widely accepted concept that 
expanded memory can always increase the potential of the 
system todo more work, even though the memory may be 
asynchronous with the CPU cycles. This is false as can be 
seen from below and has lead to some very expensive 
nistakes. The severity of the problem is related to the 
difference in cycle time of the memory and the cycle time of 
the CPU. Atypical example of this is found in the IBM LCS 
on a IBM 360/65. The cycle time for the LCS is 8 microse- 
conds compared to 1 microsecond for the CPU. Asynchronous 


devices that mus* work together, must get in synchronization 
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with each other before anyting can be done. This means the 
faster device, usually the CPU, must wait for the slower 
device, in this case, the memory. Cache memory helps 


relieve some of the delay and speeds up memory  I/O 
processing. In order to achieve synchronization, some forn 
of "hand shaking" is necessary to allow each device to 
communicate efficiently and correctly with another device 
and assures that both are ready to communicate. In the IBM 
360/65 an inhibit pulse can be easily monitored by a hard- 
ware monitor. This pulse from tns LCS travels t9 the CPU 
and stops the actual execution of instructions. However, 
"the meter continues t mone, the Wait light stays out, 
accounting time continues, but no "znstructions can be 
executed. In this systen, the inhibit time can easily rise 
to 30* of tctal elapsed time. In essence, the CPU is halted 
30% of the time. 

Today's computer systems are a collection of asychronous 
components including memory, CPU and channels. The fastest 
of these must always wait for Tha slower: one to finish. 
Devices that are most closely match2d in cycle speed to the 
CPU are more likely to have less negative impact from the 
Ennhespic. Thererore, the intuitive conclusion of adding more 
memory to improve performance is not necessarily true, and 
ence again, careful measurements should be made to examine 
the amount of time spent in the inhibit state. 

5. Fallacy: Economies of scale or "Grosch's law" still 
apply in computing. 

This intuitive ccmment may have been more popular prior 
*o the advent of minicomputers. Today, this may not be such 
a wide spread feeling since costs of hardware have dropped 
tremendously with the additional advantage of increased 
capability (CPU speed, amount of main memory, 2tC.). Herb 
Grosch, of IBM and past president of ACM, inferred that the 


capacity or power of a computer increased as the square of 
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the cost of the computer. A study referred to in Carlson 
(Ref. 17], indicated that minicomputers can be up tn 300 
times more cost effective (cost/million matrix muitiplica- 
Gomes anstmuctions: high of $5,102 for IBM 360/30 and low of 
$16 for Data Seneral Eclipse from) than the older mainframes 
or maxi computers. 

6. Fallacy: In virtual storage systems you can get by 

with less real memory than before. 

Although virtual memory does simplify many problems 
associated with multi-task systems, it has not been clearly 
established that less memory is raquired. Ina logical 
sense,  *he user can ke lead to believe that they need less 
real memory in a virtual system. Although the evidence is 
not totally conclusive, the conclusion reached in the August 
and September 1974 issues of EDP Performance Review on VS 
performance tend to indicate that additional real memory is 
actually required in crder to maintain the r2-virtual 


memory system performance level. 


B. SOME POSSIBLE FALLACIES 


Meise or possible fallacies include: 

1. Possible fallacy: The use of double or multiple buff- 
ering or a multiprogrammed computer is better than 
Single buffering. 

According to Carlson, these does not seem to be a clear 
one-to-one relation between additional memory and channel 
tie up in terms of what's best for the total system 
puroughput. Me Ua lon 1S5 In short bursts on a nulti- 
processing system then the system can often switch with 
relatively little overhead to other processing while the 
transfer of data is going on. Adequate data to prove or 
disprove this intuition is obtainable through the use of 


benchmark program runs with various buffer sizes, combined 
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٣... ۲00860779210 of the total system activity ead ths‏ -ٗگ 
Foral job throughput. ۱‏ 
Possible Fallacy: More of a primary resource (main‏ .2 
nenory, disks, tape drives, channels, etc.) will‏ 
Speed up a system.‏ 

Very often it is assumed chat, nG. OE certain 
resources will improve system performance. Ne have seen 
earlier, this is not necessarily true for asynchronous main 
memory. 

3. Possible Fallacy: If no one is complaining, all is 
well. 

It is rare that people stay satisfied about something 
for very long. At times, programmers and users may "accept" 
things with the attitude that its! the best the system can 
do, even though, say, the response time desired is not 
currently being supplied by the System. Users and progran- 
mers should ba encouraged to freely provide and make sugges- 
tions and make known their concerns and comments about 
system performance at all times. 

4. Possible Fallacy: Assembly language is faster and 
costs less then high level languages like COBOL. 

Assembly language could be faster (in execution ser 
special cases and when used in the critical secton cf a 
Programs, or routine. In general though, nedium to large 
programs written in assembly language are not "faster" and 
less expensive then a high level language when the fcllowing 
considerations are involved; 

a) Coding time. 

b) Human debug time. 

c) Machine debug time. 

d) Compiler/assembly time. 

e) Execution time; and frequency of execution. 
f) Program maintenance time and cost. 
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C. GENERAL FALLACIES 


A wide range of cases are covered by the ‘following 
general fallacies: 

1. Fallacies of generalization. 

This fallacy is developed when using statistical 
sampling and neasuring one event than generalializinge about 
apparently related events as wall. One example is the use 
of CPU busy measurments to make inferences abcut other parts 
or the system or the System in general. 

2. The fallacy of over-reducing complexity to 
Simplicity. 

This is inversely related to the previous possible 
fallacy. If a complex problem is overly reduced to a simple 
one, then the information obtained or the solution found may 
only apply to the simple problem and in no way be relevant 
to the complex problem. ^ Although at times very desirable, 
sinplification of prcblems must be done very carefully and 
not be done entirely to reduce the detailed level of data 
measurement and collection. 

3. The Fallacy of composition. 

This is tae assumption that because one item of a group 
has a particular property, the whole group has the sane 
property. This can cccur, for example, if only one of many 
terminals or disk units is measured and it is assumed that 
all others in the qrcup behave in the same way. Again, the 
data collected can usually only be used to describe the 
behavior of the specific device that produced the data. 

4. The Fallacy of appeal to authority. 

Most people are exposed to this fallacy in early life 
and live with it throughout their lives. Always check out 
the validity and extent to which someone is an "authority". 
If a vendor or product salesmen makes a claim, or statement, 


or suggests a solution to a problem, request the grounds 
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upor which the statement was made. Check “2: sau. DY 
contacting other technical people char are directly invcived 
Moth the product or method. For example, a system 
programmer may suggest that the system needs more ain 
memory so that programs may not have to be "overlayed" in 
erder to execute. This may help the programmers or ease 
their problem, but could have no, or even a negative effaec- 
on the system in general. Appeal t» authority is one of the 
"Cheapest", quickest, and easiest methods to solve a problem 
he infomation Given is correct, but it can be the most 
dangerous way, 1f the in omia tion is incorrect. Taerezore 
it could prove to be the most expensive solution. 

The next secticn discusses a suggested performance 
imporvement precedure that will hopefully help avoid the 
previously mentioned fallacies through the use of hypothesis 
Zormulation and verifiction. 


D. A SYSTEM PERFORMANCE IMPROVEMENT PROCZDURE 


Sel Re. 3], 12 “GOntrast 9 Figure 2.1 "A common 
approach to performance efforts", Suggests a seven phase 
testing procedure depicted in Pigurz 6.1 and listed below. 

1. Phase-1: Understand the System. This is part of the 
CPM effort in terms of management organization of the 
mucsiation, e description asd characteristics of the 
workload on the system, descriptions of the hardware 
configurations and software programs, etc. 

2. Phase-2: Analyze Operations. This phase also 
involves CPM and collects more detailed information 
in order to assess where bottienecks may exist and 


how well the overall system is "running". 


3.  Phase-3: Formulate performance Improvement 
Hypotheses. Formulate specific performance-oriented 


hypotheses about causes and solutions to problems and 


bottlenecks uncovered in Phase 2. 
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Figure 6.1 Recommended Performance Improvement Procedure. 


a. 


Phase-4; Analyzz Probable Cost-Effectiveness of 
Improvement Modifications. ry to establish if the 
result will justify the investment required by the 
effort. 

Phase-5: Test Specific Hypothesis. This phass 
usually involves the largəst effort in the CPE 
process. Figure 6.2 from B211 (Ref. 3], provides an 


orderly sequence of steps relevant to validate the 


hypothesis. 

Pbase-6: Implement Appropriate Combinations of 
Modifications. It may be worth while to consider 
performing several modifications at once as opposed 
to doing them one at a time in order to save "down 


83 












Formulate Choose 
















Testable Tool, Design Data Hypotheses 
Hypotheses Tests 





Adequate 
Test ? 









General 
Hypotheses 
Valid? 


Refarmulate Improvement Hypotheses No 


Implement Modifications 


Él A تھے‎  ح‎ Ee ee re es ee ee 


ہس A A ee‏ کے یی کسی کے .کے سے e KO‏ مت ہے د ید کے سے جت کت is‏ ج ha amo‏ 


| 





Figure 6.2 Suggested Hypothesis Testing Procedure. 


time" or overhead generated by making a single modi- 

fication.: Analyze what impact @ composite modifica- 

tion may have on the system and its users. Will 

programs have to be modified and réconpiled? 

7. Phase-7: Test Effectiveness of Modifications-Collec* 

data to establish the effec: the modification had on 

the performance of the system. If appropriate go to 
phase-3. 

Bell provides valuable insite that is still relevent and 
applicable to day in the following areas: workload,  opera- 
tional characteristics, and system characteristics. In each 
area he lists several questions that should ba asked. 
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In 


üoscr:bes 


contrast to the fallacies above, 


several hypotheses relating +o 


methods he proposes fcr system performance 


three basic methods along with soma of the 


1. 


Reduce Workload 


a) Hypothesis: A Free-Good Approach Has Lead to Large 
Demands. This basically means tha- i£ use of the 
computer is free then users will produce many 
programs that will taka advantage of this and 
possibly overload the system with unnecessary 
workload. 

b) Hypothesis: Unconverted Workload das Caused 
Inefficient System Use. System upgrades usually 


involve new and 
(1/2, 


eco.) 


more efficient 
new language featur 
However, many pro 


take advantage of new features 
inefficient practices. 

Tuning the Systen 

I/O Contention for 


a) Hypothesis: 


Slows Processing. This can be a 
too much data or too 
one disk pack or channel. 

Upgrading the Computer Systea 

The 

Upgraded to a Multiprocessor. 

of 


adding another CPU. 


a) Hypothesis: Computer 


power a machine may help 
If 
are adequate for multiple CPUs, 
effective ncve. 

The 


b) Hypothesis: 


Ways to 
es such as 


grams are not 


System 


in the short 


Current System Must 


do tıi.nge 
FORTRAN 77, 
88 6 © 


ande contenue old 


a Specific Device 


result of placing 


many conflicting programs on 


Should Be 


Increasing the CPU 


run Dy 


other perpherial devices 
this can be a cost 


Be Replaced 


With a Larger System. This can result from the 
workload outgrowing several components of the 
system not just the CPU as in the previous 
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hypothesis. Upgrading within the same computer 
"family" helps to reduce conversion costs and may 
be the most cost-effectiv2. However, caution mus: 
be taken and a review of alternative machines 
shculd De considered as well. A trade-off of =asy 
conversion for a modest in stead of major incrsase 
in CPU speed may not be justified. 


E. SOME POSITIVE GUIDELINES 


iD 
Qu 
Ph 
H 
0 
E3 


The following axioms found in (Ref. 17], wera adapt 
EES "Historical Fallacies" (Ref. 18]. 

1. Questions relating to performance issues must be 
operational. That is, they can be answered with 
factual evidence (supportabl2 by measurements). 

2. The questions should be open-ended to allow explora- 
tion, but they should not bs so wide open that thers 
p-cuncodz2recticn. 

3. The questions should be flexible so we can adap-* than 
zo new information. There should be no early hard- 
ening of the categories. 

4. The questions must be analytical. This means they 
can ba broken down into smaller parts so we can 
obtain answers to manageable pieces. 

5. The questions must be both explicit and precise so 
that others can truly understand what we're trying to 
get at and can concur with the answers found. 

6. The questions must be tested against actual operating 
Systems. 

Commcn sense is a good tocl to use. However, as previ- 
ously shown it can easily lead to fallacies of intuition. To 
use this tool alone would be foolish and possibly wasteful. 
Therefore, common sense should be used to develop a 
hypothesis, backed up by precise conclusive measurements. 
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Mach each collection of performanca ñata, a simple binary 
choice is not usually avaliabie. TAS ead of jast one fork 
in the road there can be an overwhalning number cf dsad-end 
enolces-easy to get cn and follow, but headed in the wrong 
direction. Performance evaluation is an extremely difficult 
task, and can be a wasteful use oí valuable resources if 
done improperly. Experience is tne best compliment to 
common sense. Both work together to increase the ability of 
the individual or group envolved in system performance 
improvements. With addition of meaningful measurements to 
Balıdate or invalidate a hypothesis or intuition, systen 


performance can be realized. 
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VII. IMPROVING PERFORMANCE WITHOUT MONITORS 

"Improving the performance of an application system 
doesn't always require hardware or software monitors, staffs 
cf specialists, nor complicated timing diagrams. Sone of 
the best results may come from just looking at the cods.", 
(Ref. 117. computer programs are usually not written under 
the most ideal circumstances by tha most ideal programmers. 
Many programmers don't become too concerned with the impact 
the program they are writing, or more correctly-their 
Meere of programming, will have In the performance of the 
system upon which the program will execute. This view is 
supported by subtle and not so subtls evidencs  conmnonly 
found in programs. For example, many programmers will need- 
lessly initialize an array to zero, even though they will 
completely fill the array with data. A less obvious example 
is the choice of data type (real vs integer) without reguard 
to the number of CEU cycles required for the alternative 
choices. Therefore, even the most efficient configurations 
of hardware and system software can still produce unsatis- 
factory performance because the applications that ars 
executed on it may not make the most effective use of the 
System resources. 

The following circumstances can have direct and/or indi- 
rect adverse effects on the performance characteristics of 


an application program or systems program [Ref. 11] : 
e Competence of the program designer. 
e The mandate given the designer. 


e The time frame under which the "program" was produced. 
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e "The reward system in effect during the writing of the 


program. 


e Conversions from one hardware or software (operating) 


system *c another. 
e The nunber of modifications mada to the systen. 


e The existance and enforcement of definitive programing 


SS andazdas and quality control. 


The list of circumstances that can effect program perfozn- 


ہے 


ance can become very dong. Thecerore, thè chances of 
improving overall system performanca through the improvement 
in individual program performanca has great potential. 
Jalics [Bef. 11] believes that, "Sometimes systems can be 


made to operate two, tehe eea ni OT fourn zines as fast with 
l1r^6rle effort". He states further, "the task of performing 
such measurement, evaluation, and enhancement is not very 
complex and does not require special skilis", 

Jaclis also feels that there are two basic 32330 9پ‎ to 
be used in performance meesurenent: 

1. "Each systen has 2 small number of critical parts 
(these critical parts system? typically comprise 15 
to 10X of all prograa statements in a systenm)." 

2. “The performance of the entire system depends toa 
large extent on the performance of its critical parts 
(that is, one can just about ignore the non-critical 
DAS 

This strategy for making the more efficient is quite simple 
and consists of finding all the critical parts of a systen, 
and modifying the areas that will have a significant impact 
on the positive performance of the systen. Jalics feels 


2In the context of this chapter the word "system" is 


defined as a. collection of jobs. Whereas jobs are defined 
as 2 col lection of one or more programs. A’ job step is one 
progran. 


89 









net ERIS Sane strategy can be applied to the whole set of 
application programs. What needs to be done therefore 6 
to concerirate on a very small part of the whole (less then 
10%) system. Desirable selection criteria, according to 
Jalics are those few application systems that: 
lí Are time-critical, that 15, systems that require 
results a very short time after input data are 
supplied. 
2. Are run frequently and consume considerable systen 
resources, or 
3. Consume extremely large amounts of computer 


resources, even if they are not run very frequently. 


A. AN APPROACH TO FINDING CRITICAL PARTS 


The best source for the determination of what jobs 
constitute the ciritical section, using the above selection 
criteria is daily and monthly job report summary reports. 
The task then, is to isolate those programs that make up the 
Critical part of the systen, find those parts of these 
programs that are executed most often and review other char- 
teristics of *he program that may contribute to it's inef- 
CIENCY Jalics provides sons guidelines to help 


accomplish this. 


1. Check Qut the compiler 


Most programs written in high level languages are 
feed as input into a program called a compiler. Most 
compilers are out of the range of modification by most 
programmers unless ttey have had compiler design and writing 
experience. However, programmers can at least learn and 
become aware of the performance or efficiency characteris- 
ecs of the Compiler. Specifically, data types and struc- 
tures and certain language features can be investigated as 
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to their degrees of efficiency. "Data type implementation 
alone can vary by a factor of 2 to 40 (one can be 40 -imes 
Gv Er chan another)" { Ref. 11 j. 

A simple method by which to look at compiler charac- 
teristics would be to write a program that contains all the 
data types and language features of interest. Af tor 
compiling the program, an assembly listing of that program 
can usually be obtained. This listing can provide informa- 
tion about the number of assembly instructions executed per 
high level program statement. OR Library routine calls, a 
machine instruction trace can be used, if avaliable. 

Experiments were run on sone of the features of a 
COBAL compiler residing on a UNIVAC 1100. ۸1۱569098 ene 
experiments were run against one specific language, it would 


ot be difficult to apply and conduct similar experiments on 
Yo 


th Ul 


oth=r languages. The following were among the fift our 
tests made by Jalics : 

1. Arithmetic statements with various combinations of 
operands--different data field sizes, various align- 
ments. 

2. Movement (I/O) of data fields with sources and desti- 
naticns of various sizes and alignments. 

٠2 Ecoudytsons using various lengths of 1*egs, align- 
ments, and use of condition-names. 

4. Table searching using loop, and search verbs, using 
Various structure tables, indices, data items, and 
alignments. 

5. Language features that examine fields using various 
lenght fields and alignments. 

The following conlusicns were drawn form the results of the 
experiments: 

1. Arithmetic operations can differ in execution by as 
much as 30 to 40 times depending upon data item type. 
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2. The most efficient data field sizes may be a mul+ipla 
of bytes and or words. The use of these siticiant 
field sizes may increase processing 2 to3 + 
faster. 

3. Alignment plays a tremendous role in efficiency of 
both arithmetic and character manipulation. 
Desirable data fields are single characters (no 
alignment required), three characters (half-word 
aligned), multiples of six characters (full-wcrd 
aligned). 

4. Avoid testing large data fields whenever possible. 
The use of condition names for more complicated 
testing offers no advantages. 

5. Built in search verbs are not always more efficient 
than a program loop depending upon the index used. 
Alignment of each table entry to a word boundary 
provides significant performance improvement. 

The input/output facilities are one of the most 
important efficiency aspects of a compiler. چ2‎ 7 
cocoa” and block sizes are a function of the characteristics 
c£ the physical storage devices, as well as the size of 
blocks most 2ffieiently handled by the operating system. 
The size of blocks can range from several hundred to several 


thousand bytes. 


2. Efficiency Measurement 





An important first step before any performance 
enhancent is to get a measure of current efficiency. What 
is needed is a way to relate the amount of useful work 
performed to the computer resources it took tə process the 
job, in order to come up with a measure of work that is 
easily understood by the users of the system and the opera- 
tions group. Examples of measure of work may be the 
۶61۱1 91: a schools registar's office may have a program 
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that does a certain amount of processing in relation to the 
Number of students enrolled in the current academic quarter 
Ge. "X" number of student records per system unit of 
tina), or a bank may do a certain amount of processing for 
each tranaction that day  ("X" number of transactions fer 
system unit of time) 

Having obtained a measure of work, anser cysnecy 
rate (ER) in terms of Units-of-Work (UOW) divided by 
Computer-Resource-Units (CRU) can be defined, [Ref. 11]. In 
the casa of a UNIVAC 1100, the job accounting system 
provided a resource unit measure as a weighted sum of CPU 
seconds and I/O seccnds CRU's can then be thought of as 
system-seconds with cne CRU equal to the complete use of the 
computer for one seccnd. Then the ER of the registar's 
record keeping program may be 54 student records per CRU, 
and the bank's ER may be 310 transactions per system-second. 

The next step is to compute the ER for every run of 
a job., This makes it possibles to compare system efficiency 
before and after performance enhancements are made, even 


though different data is used in each run. 
Sg mEGEEaSnipg the 2٤1212727 Untts 


It is highly desirable +o compute and make available 
wO ER at the end of the execution of a job. This 2mEorusg- 
tion can be placed ina file for review by interested 
personnel throughout the day ard thən tabulated into a daily 
ER/jok report or summary. 

If the job consists of only one program, calculating 
the ER is straight fcrward, since the UOW and the CRU are 
easily obtained. This task becomes somewhat more involved 
with jobs containing multiple programs in that UOW and CRU 
data will have to be collected and summed at the end of the 
dob. 


9.3 














If multiple job steps are anvolved, computer usage 
should be captured at the end of each job step or progran. 
Two sources of this information are: the operating system 
may automatically put this information in a log or there may 
exist job control language commands or supervisory calls 
that will make avaliable the CRU information about tne job 
step. Having this information about all programs that make 
Ua CE helps to identify the critical job steps or 

rograms. 


Having run the system several times, and identified 
the critical job steps, a similar procedure is used to 
locate the critical sections within a job step or program. 
Once identified, the critical sections of a job step should 
be looked at using a compilation listing of the program with 


a cross-reference index of all data names. 


ct 


6. une e Program and Compare the Results 


The next section gives specific information on 


tuning a progran. The critical section should be analyzed 
for inefficient code or practices and modifications 
suggested to correct these. After having implemented all 


modifications to the critical section of a job, using tuning 
techniques similar to those in the next section, the above 
review process should be started anew. Different areas may 
now become critical job steps as a result o£ modifications 
EO ebe mold" ertical section. This process should be 
repeated as long as it felt that the rusults justify the 
efforts, which is a decision unique to every situation. 
Guidelines on making this decision should be incorporated in 
the CPM plan of the facility, as well as in the individual 
GPE effort. 
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B. PROGRAM TUNING HINTS 


rogram tuning is within the capability cf nos? progran- 


mers Once they have an idea of what it is about. AS 
mentioned  sarlier, there is potential for significan 
program performance improvement, than Zusm can lead te 


overall computer performance improvement. 


Jalics (Ref. 11]. provides list of do's and don'ts 


along with ideas related to program tuning practices. The 


following were obtained from this list: 


1. 


Find the intermost loops in the program. Å visibly 
well constructed program may give some hint as to 
what code may be executed nost often. Since the 
contents of a data variable may détermine the 
activity of a locp, pregrammers could write a progran 
using hooks to identify those parts of the job step 
or program that are most active. These parts usually 


involve loops or innerloops. 


Next, move all superfluous statements outside of the 
critical section. Such statements include initial- 
imma constants, initializing records to zero or 
blanks even though they will be filled with new data 
each time, initializing whol arrays to zero when not 
necessary, etc. 

Try to locate inefficient uses of data types and 
convert them to the most effiecient type within the 
intended precision, etc. 

Investigate data field sizes. Different computers 
may have different optimum size boundaries such as 
quarter word, halfwerd or full word. For character 
data, it may be more efficient in teras of processing 
overhead, to use storage fields that are multiples of 
words to avoid alignment overhead associated with 


parto cal word fields. The overhead associated with 
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alignment  beccmes much mors obvious when irside a 
loop that nay have several thousand iterations. 

In complex conditional Statenents, test for what is 
most probable first. In the case of an IF statement 
with multiple conditions, place those ccnditions 
ie OE he 


first that are likely to decide the res 
2 the condi- 


t & 


ccmpcund condition without testing al 
tions. When conditions are ORed, the condition most 
likely to bs true should be checked first, 7 
ANDing, the condition most likaly to be false should 
be checked first. fhe 3I kolchooq'^ information 
required in the above conditionals is not always 
known, but when it is know it may prove very worth 
while to make an effort to take advantage of it. 

In a Situation involving a series of consecutive IF 
statements, the IF statement most likely to be true 
should come first then the next most likely second, 
SIG. 

Avoid testing large data fields. For example, it may 
be necessary to load a large data field with all 
zeros or blanks but dosn't directly check the whole 
field to see if it is indeed all zeros or all blanks. 
Rather, use a "flag" (a small status field) or set a 
bit in the field indicating the fields zero or 
nenzero status. Checking a few bits over several 
words may save considerable time. This method should 
be used for any large data structure when appro- 
Dear te. 

If possible avoid the use of verbs that are very 
time-consuming such as COBOL's TRANSFORM, INSPECT and 
EXAMINE whenever possible. One example inJalics 
(Ref. 11], used EXAMINE on a record with 932 charac- 
ters which resulted in 6,127 instructions being 


executed. 
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10. 


Beware of language interfaces and subroutine call 
overhead. Overhead associated with certain language 
invoked constructs can vary significantly. In COBOL 
for example the efficiency of the PERFORM verb versus 
the CALL verb should vary greately depending upor 
whether internal subroutires are used or external 
subroutines are used. Upon every subroutine call, 
some compilers will generate code that will dynani- 
cally obtain mere memory to save the environment of 
the calling routine. Upon subprogram termination, 
the memory is then released. According <0 Jalics 
Some compilers have little known options that will 
1s 
reuse it each time the subprogram is called. A 


either acquire additional memory dynamically oc 


(o 


(D 


sinple exanple which can be very inefficient in terms 
of time and space involves calling a subroutine to 
compute a value based upon the contents of a field in 
5 ع۶‎ 9> Many consecutive records processed may 
have the same contents in the field and a subroutine 
is called each time to compute the same value. It 
would be much more efficient to retain the contents 
of the field and computed value of the last record 
processed and compare it with the current record and 
only compute anew value by calling the subroutine 
when they differ. 


(D 


Know the options available on the compiler. Son 
compilers have options that provide several levels of 
effiency. Some compilers have memory efficient or 
execute time efficient opticns. Code that executes 
inline may take up more menory but execute faster 
*hen code placed in routines and called, which uses 
less memory. Many other options are available that 
may range check data structures and subscripts, place 
unnecessary code on the outside of loops, etc. 
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11. 


12. 


13. 


14. 


However, some of these options are intended to ba 
3--017 ۹0۰320097 36 final compile, and 1£ tusned on 
during program development can waste significant 
amounts of coapile time. Therefore, constant use of 
inappropriate compiler options during all phases of 
program develcpment may have an adverse impact on 
overall system performance. 

Take a close look at tables or arrays. Certain 
alignment characteristics nay make the compiler 
generate efficient code for arrays and tables. This 
is true in Machines that are "word oriented and 
operate much more efficiently on tables where sach 
entry is word aligned. This is a trade off between 
time and space, since tables elements mav have to be 
padded to take advantage of the alignment efficiency. 
Look at the subscripts used for addressing tables. 
In COBOL for instance, subscripting can be more effi- 
cient than INDEXING. 

Look at table search technigues. In certain cases 
ndezing (ie. Storing xxx in location xxx) can be 
much more efficient then simply searching a table 
(Ref. 11]. The indexing method is very direct. Tr 
searching is required make sure the most efficent 
method is used. This may iepend upon the order or 
arrangement of the data within the data structure. 
In some cases, it may be more efficient to order the 
data to achieve better search efficieny. 

Is the current read requesting the same data as the 
last read? It may be worthwhile to check the current 
read request to see if it is requesting the same 
information as the last read. Some applications may 
have a high cccurence of this, in which case many 
unnecessary I/O transactions can be avoided. 


98 





15% 


T6. 


17: 


Review the processing technique of a progran, is it 
the most efficient? Random access techniques allow 
the processing of records in any order. However, as 
mentioned earlier, ordering the data in the sequence 
in which it will be processei could cut down on disk 
access overhead time in large files, since the data 
would hopefully be stored to provide consecutive, 


inline disk accesses. 


Is file organization optimal? Three commonly known 
file organizaticns ara sequential, ind=ex=d- 
sequential, and direct. Phe CHOICE Of Organizer loz 


ased for a file for a particular application can 
substantially impact performance of the whole systen. 
Look at file blocking buffer size. Tae programmer 
that is aware of the size of physical blocks which 
are efficiently processed by the operating system, 
the access method, and tha various storage devices 
will be better able to produce efficient  I/O. 
Programmers that are unaware of these are more apt to 
write inefficient code and negatively impact perform- 
ance. Block sizes on a disk usually correspond to a 
sector and the block size for an operating system is 
constrained by how much memory was allocated for the 
I/O buffer. 
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VIII. CURRENT AND 


FUTURE TECHNOLOGY AND PERFORMANCE 

Increasing end-user computer power, micro computers, 
networks, data base machines and database management 
systems, multiprecessor chips, multiprocessor computers and 
fifth generation computers are all relatively new fislds and 
technologies that will most likely demand some type of 
performance evaluaticn. This chapter will explore some of 
these technologies in terms of performance evaluation. New 
performance parameters will have to be defined as well as 
tools and methods to measure then. The research literature 
avaliable in this area is limited. 


A. END-USER COMPUTER POWER 


As a result of the favorable economic trend in the 
computer hardware industry, complete computers with respec- 
tible computer power can reside on less than one halt of the 
eyptcal office desk. Performanse issues in this area 
concern the capacity and reliability of the communication 
linkage between the main computer system and the user equip- 
ment; the adaptability of this end-user equipment to the 
various experience levels of the group of users: and the 
amount of conversion of non-standard equipment and software 
required in order to be compatabi> with the main computer 


system. 


B. MICROCOMPUTER PERFORMANCE 


It is not long after acquisition that the microcomputer 
user realizes the potential of senling or receiving infor- 
Bation or data to or from other microcomputer users. This 
involves communication lines (usually existing tslephone 


100 








lines), modems and compatability-the ability cf the 
computers to understand what each is sending +o the other. 
If we consider the interaction of the user with the niczro- 
computer as being part of the overall systen, then user- 
machine and user-scftware interface "friendliness" can 


impact perfornance of the systan. 


As discussed in an earlier chapter, the rete at 
which data or information is passed through a mcdem can 
severely impact the performance of the computer. Use of a 
slow, 300 baud modem between two systems can produce the 


same performance as if poor terminal response time and 


system slow down due to a heavy workload. The soluticn to 
this problem is simple but expensive. The user should 
purchase the fastest modem affordable. Unfortunately, the 


price of 1200 baud modems is considerably more than the 
slower 300 baud modems, and most nome microcomputer users 
have 300 baud modems. 


2. Compatability 


The other performance issue requardina microcon- 
puters that communicate with each other is compatability. 
If two micrccomputers are incompatable and can't communicate 
directly with each other, ther conversion or interface 
programs/hardware are needed. This adds overhead and 
reduces the performance of each computer since time must be 
spent converting the data or information to a format which 
is understandable by computer. A standard interface chip 
placed in each communicating computer could be a solution 
to this problem. The issue of incompatability becomes a 
night-mare when a large organization like an insurance 
company allows uncontrolled proliferation of microcomputers 
and then decides tc tie all departments together into a 
computer network within the organization. 
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3. User Interface 


0 
u 


The microcomputer has exposed the computer to 


nas 
of users. Many of these users simply can not type and ar 


(D 


frustrated by the use of keyboards that require some degre 


T 


of finger dexterity. Many interactive programs and packages 
require typed responses, which result in reduced system 
perfcrmance when used by this class of users. Therefore, 
alternative devices are required to help improve the inter- 
action of the user with the computer. These devices do 
exist and are called "picks" and include the mouse, the 
EU pen, the joy stick and the touch panel. Touch panels 
seem to have the mest reliability since they involve the 
users finger Ir a pencil and a grid system that detects the 
location of the pointing device relative to its position on 
the Screen. Therefore, the user simply points to an item in 
a nenue that provides the user with a choice of responses. 
Newer systems use touch panels together with icons-words or 


ideas represented by an image. 
4. Inexpensive Software 


Since so many micrccomputers have been sold, “the 
Gemand for software, expecially inexpensive software is 
great. This tends to produce both application and systen 
software that may not be performance conscious. Cheap soft- 
ware may run but it may run very slow and use excessive main 
memory. Therefore, software should not be purchased by price 


alone but by its reputation if possible. 


C. NETWORKS 


Telecommunications and networks are becoming very 
popular and should see wide spread use in the 1980's. 
Designers of these technologies are providing a great varity 
of choices in the area of data carriers and data termina- 


EOL S: 
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e Front-erd processors 


PBX (private branch exchange) 


e Modems 


Voice satellite links 


Performance evoluticn in this area has been hampered by 
alnost daily changes in communication technology. 

Network design, development and testing tools cover a 
broad range of analog, digital, and protocol analyses. The 
physical components that these areas deal with are: carrier 
lines, modems, and other forms of communication. Because of 
the potential size cf networks (several *housand nodes), 
only software tools can cover the ovarall giobal nature of 
the network. These software tools help network designers 
plan, analyze and optimize tha performance end cost of the 
system. Analysis program software gathers message traffic 
information, measures message response times, determines the 
probability of blocked messages, and ultimately defines the 
maximum traffic volume the system can carry within accep- 
table response times. 

Test objectives of protocols include overall evaluation 
of the software and control sequences from one end of the 
System to the other end of the system. On a character-by- 
character basis, timing signals and properly controlled 
synchronization are reviewed as well as error detection in 
the following three categories: characters, Cod ng; sand 
parity. 

A central data base is created and used at the systen 
level design of a network. Various programs, {using paran- 
eter information from the data base), model many possible 
configurations and analyze them in terms of their 
performance and cost. The sheer complexity of data commun” 


ciation networks (sone may eventually have 7000 on-line 
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units) necessitates software monitors. This softwareis 
supplied by the vendors of front-end processors (e.g., tas 
IBM 3705 communicaticns processor). The programs analyze 
the data traffic controlled by the front-end processor and 
measure such information as the number of busy signals and 
the number of nonresponding units. The advantage of vendor 
supplied programs is that they have are able to access all 
communication lines controlled by the processor. In 
contrast, protocol analyzers and other test hardware exanine 
only one line at a time. 

Netwcrk performance can be hampered by failed stations 
or transmission components. Failure analysis software exan- 
ines the avaliable channels and the expected number of 
failed devices per operating period. This is done by calcu- 
lating the effectiveness of the fault-diagnostic, fault- 
control, and outage-prevention measures that are built into 
the system. Expected transmission errors in the connection 
links themselves are studied and added to the expectation 
of overall system performance. 

By exanining the potential data traffic capabilities and 
subtracting the effects of blocked messages, faults, and so 
26 , the designer can measure the performance of the 
system with regard to thoughput (bits or characters per 
second), messages per interval, and message response. Given 
information on traffic variations, the system's sensitivity 
to volume can be projected on an hourly, daily, and annual 
basis. If a given configuration meets the raw thoughput, 
traffic volume sensitivity, and reliability factors, its 
construction can be optimized to iower costs. 

All of the system's performance measurements can not be 
optimized Simultaneously. It is in these instances that 
performance analysis software proves IS WOT th. The 
programs indicate the tradeoffs among various components and 
methods, which allows cost-effective system to be designed 
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With a reasonable response time and with minimun performance 
Be@eadation in the face of violations in traffic, erro 
Gace, Of failure rate. 

Data communication tests includeanalog tests on the 
carrier lire and digital tests on the interface etween 
modams and data terminal equipment (computers and terni- 
nals). The protocol-level tests a3valuate data and cortrol 
sequences Character by character. See figure 8.1 for a 
graphical representation of the data communications testing 


environment. 
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Figure 8.1 Data Communications Testing Environment. 
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Existing telerhone lines are used by most networks 
due to tbe low cost and wide spreal availabili-y. Ho 
these lines are not designed for tha high speed, error fires 
transmissions desired by the network and therefore cause 
problems. DIerors  Chde can afzece the performance of a 
communications line are analog parameters covering  band- 
width, transmission node, and transmission problems. 
Therefore, the analeg problems reduces itself tc testing 
individual lines and determining if the line meets the 
network requirements and specifications. TIMS (Transmission 
Impariment Sets) can provide comprehensive tests and meas- 
urements of an analog line. Measuraments attainable by this 
device are noise, the gain slope, noise-to-ground and 
signal-to-noise ratios, and peak-to-average ratio. The 
peak-to-average ratio is a quick benchmark measurement and 
"gagues the overall spreading or smearing in time cf a 
Signal transmitted across the communication channel" 
according to Bailey [Ref. 19]. 

Measurement and testing of the carrier line is best 
accomplished end-to-end by tramsmitting a test signal at one 
end to a meter at the other end to receive and measure it. 
This simple method will indicate which end, transmitting or 
recieving, has a prcblen. If the communications line is 
found to be reliable, then the problem must be in either the 
modem or the DTE (data terminal equipment). 


Digital testing is required to verify the data being 
transmitted and received over the lines. By sending a fixed 
pattern test packet through the system and using a data 
Simulator or protocol analyzer in place of certain systen 


components, data transmissicn errors can be identified. Ir 
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the data becomes suspect, the mcdem can be checked w 


Baar of BERT's (Bit Error Rate Tsstors). One BERT serds 


rt 
> 
m 


$1 


data sequence to the transmitting modem and another BERT 
analyzes the data as it passes through the receiving ۰ 
A bit error ratio can be established and is a good measure 
of overall link performance. A typical voice communications 
link should have a ratio nct exceeding one error in fifty 
thousand bits, according to Bailey. Measurements capability 
cf BERT'S vary with manufacturer and include parity check, 


Block error rates, carrier loss, cock slsDS, ard timing 
7177526 ۶9655. The distribution of ezrors ovaor a given bit 
straam is called block error rate. Throughput of a systen 


mia] ss capable of block=-error detection recognition is 
defined as the ratio of error-fras blocks to total blocks 
transmi-ted. Blocks that are found to contain errors must 
be retransmitted. This impacts ‘throughput rate, which in 
turn impacts performance of the system. A large bit error 
Rc sr conjunction with a low block er-or rate would 
provide Letter performance than an equivalent bit error rate 
distributed evenly across the data stream. 


3. General Performance Summary 


E 


Problems such as carrier loss, clock slippage, and 
timing distortions warn the designer that problems exist 
within *he analog carrier line. The proper operation of 
modems and carrier lines can be verified, with the aide of 4 
good kil error rate tester, through the use of an algorithm 
that isolates problems to various components of the data 
communications link, see figure 8.2. 

The next step is testing the DTE, which uses data 
simulation, data monitoring, and 52690691 analysis. 
Protocol analyzers have the same capabilities as data-link 
monitors and simulators, as well as their own data analysis 


Sanctions. 
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When serving as a monitor, the protocoi analyzer is 
attached to the interface between the moden and she OTE, 
which may be 2 CPU or a terminal. No overhead is incurred 


by the operating equipment from the data analyzer because 


the analyzer is connected in parallel. Once the analyzer 
captures the transmitted data, the operator observes the 
data and protocol characteristics to find errors. He then 


either freezes the frames to catch the data or operates at 
slow speeds. 
Functions of the equipment at either end of e 


th 
channel are duplicated by the analyzer when it is in th 


(D 


simulation mode. It tests the part of <he data communica- 
memos 27nK “that ¿is dewn or generates errors that test the 
ay Of CPUS or terminals to recover from transmissicn or 
protocol errors. The instrument can also test new equipment 
before it is added to an existing network or laboratory 


system. 


4. Ferformance Checking Intel Network 


The NDS-II is an Ethernet-based network under devel- 
opment by Intel. Performance issues of the network are 
exaimed using hardware tools. A nunber of Intellec develop- 
ment systems are tied together by a resource manager, which 
handles disk storage and files, <o form a resource-sharinag 
network of develcpment stations. A plugin controller board, 
Bn Contains an Ethernet VLSI chip set and an 8- or 16-bit 
processor, connects each station to the Ethernet network 
(Ref. 19]. 

Data paths are analyzed by two different hardware 
monitors in the NDS-II systen. One monitor, a NCR Comten 
Dynaprobe data  monitcr and analyzer, is capable of moni- 
toring data lines of up to 10 MHz with  10-ns resolution. 
Signal levels and tranctions are counted using the instru- 


ment's 16 counter-timer probes when attached to a data 
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communcication channel. This monitor measures the time each 
I/O or disk channel spends in various activities. 

The second hardware monitor is 1ہع چ8‎ 77 
performance analysis tool, which measures CPU activiites. 


It is somewhat similar to an in-circuit emulator, but only 


acts as a passive data monitor. It is composed of  *wo 
Multibus boards and a buffer board. The processor to be 
tested is removed  frcm it's slot. The buffer board is 


placed into the socket connectors of the processor  *o be 
tested, which is pluged into the burfer board. This =stab- 
lishes the analysis  *ool in a posicton to monitor the CPU's 
address, data, conc olaaa status laes and count ths 
number of address hits. When a trigger is tripped, it can 
record the following information: reads and writes to memory 
or I/O, operational code fetches, and, when monitoring an 
8086, cp code execution. 

Intel, through the use of both analyzers, was able 
to pinpoint bottlenecks and *o increase the number of 


Supportsd users per link from 8 users to 16 users. 


D. DATA BASE MANAGEMENT SYSTEM (DBAS) PERFORMANCE 


DBMSs are experiencing a constant increase in popularity 
and use. À DBMS isa special purpose, complex software/ 
hardware system, requiring dedicated resources of its own, 
plus substantial resources from the host computer system. 
DBMS performance can have a significant impact in overall 
hos- computer system performance. The design of DBMS bench- 
marks and the interpretation of measurement data is 
presented in a paper "Experiments in Benchmarking Relational 
Database Machines" at the Third International Workshop on 
Database Machines [Ref. 20]. 
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Aitken [Ref. 21], describes a DBMS model char 
provides performance evaluation ani prediction information 
and can be used to: 


e Predict the performanca of a DBMS for a given data pase 
design and DBMS workload. 


e Investigate DBMS performance relative to changing work- 


load, and perform DEMS stress-test studies. 


e Evaluate alternative logical and physical data base 
designs to achieve a reasonable p2rformance-efficient data 


base design. 


e Support DBMS performance tuning and investigate  DBMS 


performance problems. 


The modeling approach results in a DBMS model which 
closely simulates the operation of a real DBMS. ECSS II 
(Extendable Computer System Simulator II), a special purpose 
language for constructing discrete-event simulation models 
of computer systems (and also a super-set of SIMSCRIPT) was 
used to implement the DBMS model. 

The model uses a simulated Data Base Control Systen 
(DBCS), which handles the user interface and manages access 
to the data base. A simulated data base appears the same as 
the real data base ccnceptually, but contains no real data. 
There exists a one-to-one correspondence between the pages 
and records in the real and in the Simulated data bases. 
DBCS operations include the following: 


e Data Base Area Management 

e Bu£fer Management--Strategy and number of buffers 

e Data Base Space Management--page overflow 

e Journaling--manages a file of before/after images of DB 
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e Data Dictionary Access 


The model uses a Data Definition Language (DDL) to 
describe a complete data base schena. A Data Manipulation 
Language (DML) is available which includes most of the oper- 
ations provided by the real DML language. 

A run-unit application program is represented proce- 
durally in the model and consists primarily of DML state- 
ments (Ref. 21]. Since the models DML statements so closely 
follow the actual DML statements, the modeling of an appli- 
caticn progran's data base activity can be relatively easily 
accomplished. 


۶2۰۰5 ء416٦‎ DEMS Performance Rela:sd Outpucts 


An automatic feature of tha DBMS model coilscts and 
reports a variety of DBMS and data base related perfcrmance 
Statistics without user intervention and are summarized 


below: 
For each area of a modeled data base: 
e queuing and utilization 
e number of pages read and written 
e page access time 
For each simulated run-unit: 
e total execution time 
e number pages requested 
e number pages read and written 
e records stored 
For each physical file: 


e number of reads and writes 
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e number of seeks and zero-time seeks 
e Seek time 
e data trensfer time 
e total access time 
For the DBMS buffers: 
e queuing and utilization 
e nunber rages requested 
e number of pages read 
e number modified pages written 
For the DBMS: 
e run-unit DML request queuing 
e run-unit concurrency level 


The DBMS model closely resembled the actual DMBS and made it 
very easy to use according to Hsu. Validation of the model 
plus the addition of more détails are currently underway. 





IX. CONCLUSION 

Ideally, every computer installation should uss Macro 
Computer Performance Evaluation--CPM and be able co deter- 
nine when and how to use Micro Computer Pperformance 
Evaluation--CPE. Idealiy, CPM and CPE would work togeéerher 
+o optonize the perfcrmance of the computer system.  Muchsel 
[Ref. 22], suggests the ultimate computer performance 
۶000 22 


000d solution to this problem is, Dno 7 a 
program we would like to call the ‘automatic opersaror! 
(AUTOP). It analyses system behavior and controls parane- 
ters to mest the operational goals of the computer installa- 
tion. It either totally regulates operation of the systen, 
cr gives advice to a human operator whenever system behav- 
iour deviates „from previously set desired values. System 
behavior is analyzed by a modified performance measurement 
program. How could one optimize system behaviour by such 2 
procedure? The methods are to control response time; to 
control CPU and I/O share of certain types of jobs, possibiy 
giving priority to  scme of them according to installation- 
specific criteria; to control system parameters according to 
tine of day and workload; to recognize bottlenecks in the 
System (and eliminate them if possible); to give warning 
when system behaviour deteriorates; and to keep humans well 
informed, reporting system utilization to human operating 
personnel and tuning group." 

Ideally, computer system designers or architects will 
take into account preblems of performance measurements and 
their interpretation when designing new systems, and manu- 


factures of computers will impliment their ideas as options 





or standard equipment in the computer system that they make. 
Ideally, software designers MAA corporate compu 
performance measurment concepts into the op=ratiing systen 
and related system software. Ideally, nore and {more 
computer facilities will use a "preventative" computer 
performance evaluation approach that will replace the "emer- 
gency Tocm" approach that most unprepared computer facili- 
ties are forced to use when a severe performance problen 
arises. 


Realistically, the currant trend of decreasing hardware 
ccsts and increasing hardware capability seem to have 
diluted the performance concerns of management in many of 
today's computer facilities. There may be a tendency, 
therefore, for computer facility management to rely soiely 
upon the "appeal tc autority”, usually the vendor 
salesman, for a solution to pərfornance problems instead of 
getting involved in a CPM and CPE effort. 

It is intended that this guide provide some reasonable 
middle ground between the ideal situation and the realistic 
Situation that the world of computer performance seems to be 
an today. In conclusion, the following steps are suggssted 
when consideriag performance of a computer system: 


e Develop a CPM Plan initially setting simple and realistic 
goals based upon the experience of the CPE personnel and 


include all phases of the computer life-cycle. 


e Implement the best and most suitable Accounting package 
affordable. 


e Locate "critical sections". 
e Tune critical secticn software. 


e Acquire mor2 sophicated CPE tools only when existing tools 
are considered inadequate and the CPE personnel are capable 
and experienced enough to use then. 
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EB USPDEOpriate, consider outside help. 
Beczeesrully, tire did not permit the applicetion and 
experimentation of some of the suggested tuning hints given 


in this guide. 
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APPENDIX A 


SOURCES OF INFORMATION 


The following list gives additional sources of informa- 
ion. EDP Performance Review annually provides an excellent 
annotated bibliography of the preceeding years' performance 
related literature. The name of the source is in bold type 


followed by address information. 


CMG Conference Proceeding 
Computer Measurement Group 
PO Box 26063 

Phoenix, AZ 85068 


CMG Transactions 
(same address as above) 


Computer 

IEEE Computer Society 
10662 Los Vaquros Circie 
Los Alamitos, CA 90720 


Computer Performance. 
Euszeryörth Scientific Ltd. 


WO Eoxa63 

Westbury House Bury Street 
Guilford, Surrey GQ2 SBH 
England 


Computing Surveys 
KC MP 3 Y 


11 West 42nd Street 
New York, NY 10036 


CPEUG Conference Proceedings 
Netional Bureau of Standards 
institute for Computer Sciences 
and 6601 

Weshington, 20234 


Datamation . 

Technical Publishing 
1301 South Grove Avé. 
Barrington, IL 60010 


ECOMA Conference Proceedings 
European Computer Measurement 
ASSOC. 

Scott Yasler 
Scheuchzerstrasse 5 

CH 8006 Zurich 

Switzerland 





EDP Performance Review 
Applied Comruter Research 
Eo Box 92 

Phoenix, AZ 85068 


=o Systems Journal 
Armonk, NY 10504 


on International Conference on 
Computer Caąapacı Taa یا ہے‎ : 
Institute for Software Engineering 
510 Oakmead Parkwa 

Sunnyvale, CA 9408 


International Sysmposius on. 
Computer Performance Modeling, 
Measurement, and Evaluation 
North Holland 

PO Box 103 

1000 AC Amsterdan 

The Netherlands 


Performance Evaluaticn 
(same as above) 


Performance Ev 
ÀACM/SIGMETRICS 
New York, NY 1 


Software Engin 
Bode Computer S 
10662 Los Vaque 
Los Alamitos, C 





A 
EXAMPLES OF 


ENDIX B 


PE 
PERFORMANCE MEASURES 


The following performance measures were obtained from 


Svobodova [Bef. 12], and are grouped according to quanity of 


service, quality of work, component utilization, underlying 


tC OTS, 


and workload characteristics. The measure is in 


bold type followed by the definition. 


12 


Quantity of work executable by a systen 


a) 


E) 


c) 


2) 


e) 


Throughput: Amount of useful work peformed by tne 
system per unit of tima (job processing capa- 
BILLIE Via 

CPU productivity: Percent of time spend in problen 
state (used as a measure of throughput). 

Capacity: Total work executaole per time unit with 
a balanced workload. 

System limits: Number of terminals that can be 
Supported cr number of jobs than can be multipro- 
grammed without serious dagradation of service. 


Reserve capability: Unus2d system capability. 


Quality of Service 


a) 


b) 


c) 


Turnaround time: Elapsed time between submitting a 
job to the system and completion of ouput. 
Response time: Elapsed time between entering the 
last character of a request at a terninal and 
receiving the first character of the response. 
For one time slice tasks Response time is the 
elapsed time to complete for tasks that require 
less than cne CPU time slice. 

Gain factor: Total active time needed to execute a 
job mix under multiprogramming/total active time 
needed to execute same mix sequentially. 
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e) 


t) 


g) 


h) 


i) 


Elapsed Tine Multiprogramming Factor (ETIP): 
Elapsed tine to complete a task under 
multiprogramming/elapsed time to complete when 
this task is the only one active in the system. 
Internal Delay Factor (IDF): Active time needed to 
complete a task under multiprogramming/active tine 
to complete when this task is the only one in the 
system. 

External Delay Factor (EDF): Task elapsed +ine/ 
task active time. 

Reliability: Probability that the system functions 
correctly at any given tine. 

Availability: Percentage of time a user can get 
system services. 

Ease of use: Time needed to prepare and debug 


programs for the systen. 


Component Utilization 


a) 


b) 


c) 


d) 


e) 


Resource utilization: Percent of time a resource 
is in use. 

System utility: Men Mee Simmons ure tization of 
system resources. 

Component overlap (device gain) : Percent of time 
operations cf two or more hardware components are 
overlapped. 

Queue length: Average queue length. 

Overhead: Percent of CPU and channel time required 
by the operating systen. 

Swap efficiency: Frequency of occurence of indi- 
vidual instruction operation codes. 


Underlying factors 


a) 


b) 


Computer power: Number of operations per unit of 
time. 
Raw speed: Actual speed of computer components. 
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c) 


d) 


e) 


Reaction time: The time that elapses from when an 


input arrives into tha system until it recieve 


(ù 


its first time slices. 

Internal response time: Time between entering the 
Nast character on a terminal and receiving first 
CPU service. 

Program efficiency: Amount of time spert executing 
individual portions of a program. 


Wcrkload Characteristics 


a) 


b) 


c) 


d) 


e) 


3) 
K) 


1) 


User interaction cycle: Time between initiation of 
successive tasks from a single terminal. 

User interaction rate: Frequency with which a 
single user request syst2m services. 

Service time (compute time): CPU time required by 
a Single task (job). 

Service rate: Rate at which requests are serviced 
by the CPU. 

Arrival rate: Rate of task arrivals to the 
processor stage. 

Terminal tine {user think-type time): Tine a task 
spends in the inactive state waiting for a user 
response. 

User intensity: CPU time requested/user think-type 
time. 

I/O times Service time at an I/O processor. 

I/O request rate: Amount of I/O service required 
by a single task. 

Active time: CPU time per task core residence. 
Blocked time: Time a task is incapable of 
receiving CPU service. 

Degree of sultiprogramming: Number of simultane- 
ously active batch users oz simultaneously active 


user terminals. 
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